2023-02-09 20:54:03,115 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure timeout: 13 mins 2023-02-09 20:54:03,342 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936 2023-02-09 20:54:03,357 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-09 20:54:03,358 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca, deleteOnExit=true 2023-02-09 20:54:03,358 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-09 20:54:03,359 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/test.cache.data in system properties and HBase conf 2023-02-09 20:54:03,359 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/hadoop.tmp.dir in system properties and HBase conf 2023-02-09 20:54:03,360 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/hadoop.log.dir in system properties and HBase conf 2023-02-09 20:54:03,360 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-09 20:54:03,361 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-09 20:54:03,361 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-09 20:54:03,494 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-09 20:54:03,905 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-09 20:54:03,909 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-09 20:54:03,910 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-09 20:54:03,910 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-09 20:54:03,910 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 20:54:03,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-09 20:54:03,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-09 20:54:03,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-09 20:54:03,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 20:54:03,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-09 20:54:03,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/nfs.dump.dir in system properties and HBase conf 2023-02-09 20:54:03,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/java.io.tmpdir in system properties and HBase conf 2023-02-09 20:54:03,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-09 20:54:03,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-09 20:54:03,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-09 20:54:04,404 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 20:54:04,408 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 20:54:04,682 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-09 20:54:04,838 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-09 20:54:04,852 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 20:54:04,888 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 20:54:04,919 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/java.io.tmpdir/Jetty_localhost_43681_hdfs____tuli0q/webapp 2023-02-09 20:54:05,070 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43681 2023-02-09 20:54:05,080 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-09 20:54:05,080 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-09 20:54:05,485 WARN [Listener at localhost/38591] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 20:54:05,581 WARN [Listener at localhost/38591] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 20:54:05,599 WARN [Listener at localhost/38591] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 20:54:05,606 INFO [Listener at localhost/38591] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 20:54:05,610 INFO [Listener at localhost/38591] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/java.io.tmpdir/Jetty_localhost_37569_datanode____xicj3v/webapp 2023-02-09 20:54:05,710 INFO [Listener at localhost/38591] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37569 2023-02-09 20:54:06,008 WARN [Listener at localhost/45961] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 20:54:06,018 WARN [Listener at localhost/45961] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 20:54:06,023 WARN [Listener at localhost/45961] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 20:54:06,025 INFO [Listener at localhost/45961] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 20:54:06,031 INFO [Listener at localhost/45961] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/java.io.tmpdir/Jetty_localhost_46257_datanode____7lq4xl/webapp 2023-02-09 20:54:06,134 INFO [Listener at localhost/45961] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46257 2023-02-09 20:54:06,144 WARN [Listener at localhost/36005] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 20:54:06,159 WARN [Listener at localhost/36005] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-09 20:54:06,162 WARN [Listener at localhost/36005] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-09 20:54:06,163 INFO [Listener at localhost/36005] log.Slf4jLog(67): jetty-6.1.26 2023-02-09 20:54:06,168 INFO [Listener at localhost/36005] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/java.io.tmpdir/Jetty_localhost_33473_datanode____.7xdisp/webapp 2023-02-09 20:54:06,270 INFO [Listener at localhost/36005] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33473 2023-02-09 20:54:06,280 WARN [Listener at localhost/45551] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-09 20:54:06,497 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x75ba5bec3d0597f1: Processing first storage report for DS-d0fad12c-c922-4412-b45d-76cca16cdd0d from datanode 5ff62a41-196c-4396-be17-ac730c5a50b4 2023-02-09 20:54:06,498 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x75ba5bec3d0597f1: from storage DS-d0fad12c-c922-4412-b45d-76cca16cdd0d node DatanodeRegistration(127.0.0.1:41149, datanodeUuid=5ff62a41-196c-4396-be17-ac730c5a50b4, infoPort=36215, infoSecurePort=0, ipcPort=45961, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,498 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52555e6844b68f04: Processing first storage report for DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002 from datanode a0ea1a13-8490-4cba-a134-da32a3f13324 2023-02-09 20:54:06,498 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52555e6844b68f04: from storage DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002 node DatanodeRegistration(127.0.0.1:44401, datanodeUuid=a0ea1a13-8490-4cba-a134-da32a3f13324, infoPort=34627, infoSecurePort=0, ipcPort=45551, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3cfdcd05a5912391: Processing first storage report for DS-36ecabf6-9ef8-4440-8798-1ec372920440 from datanode a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3cfdcd05a5912391: from storage DS-36ecabf6-9ef8-4440-8798-1ec372920440 node DatanodeRegistration(127.0.0.1:40557, datanodeUuid=a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa, infoPort=34661, infoSecurePort=0, ipcPort=36005, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x75ba5bec3d0597f1: Processing first storage report for DS-a7a552ab-77c8-4221-acb2-5fb8da11b3f0 from datanode 5ff62a41-196c-4396-be17-ac730c5a50b4 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x75ba5bec3d0597f1: from storage DS-a7a552ab-77c8-4221-acb2-5fb8da11b3f0 node DatanodeRegistration(127.0.0.1:41149, datanodeUuid=5ff62a41-196c-4396-be17-ac730c5a50b4, infoPort=36215, infoSecurePort=0, ipcPort=45961, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x52555e6844b68f04: Processing first storage report for DS-f7b824e5-b46e-406e-be96-af86f75e7ea1 from datanode a0ea1a13-8490-4cba-a134-da32a3f13324 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x52555e6844b68f04: from storage DS-f7b824e5-b46e-406e-be96-af86f75e7ea1 node DatanodeRegistration(127.0.0.1:44401, datanodeUuid=a0ea1a13-8490-4cba-a134-da32a3f13324, infoPort=34627, infoSecurePort=0, ipcPort=45551, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x3cfdcd05a5912391: Processing first storage report for DS-e3914fbb-dd53-4148-ae61-6e8424a31e84 from datanode a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa 2023-02-09 20:54:06,499 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x3cfdcd05a5912391: from storage DS-e3914fbb-dd53-4148-ae61-6e8424a31e84 node DatanodeRegistration(127.0.0.1:40557, datanodeUuid=a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa, infoPort=34661, infoSecurePort=0, ipcPort=36005, storageInfo=lv=-57;cid=testClusterID;nsid=818402334;c=1675976044485), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-09 20:54:06,689 DEBUG [Listener at localhost/45551] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936 2023-02-09 20:54:06,752 INFO [Listener at localhost/45551] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/zookeeper_0, clientPort=58959, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-09 20:54:06,767 INFO [Listener at localhost/45551] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58959 2023-02-09 20:54:06,774 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:06,776 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:07,433 INFO [Listener at localhost/45551] util.FSUtils(479): Created version file at hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686 with version=8 2023-02-09 20:54:07,433 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/hbase-staging 2023-02-09 20:54:07,726 INFO [Listener at localhost/45551] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-09 20:54:08,180 INFO [Listener at localhost/45551] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-02-09 20:54:08,211 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,212 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,212 INFO [Listener at localhost/45551] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 20:54:08,212 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,213 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 20:54:08,351 INFO [Listener at localhost/45551] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 20:54:08,421 DEBUG [Listener at localhost/45551] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-09 20:54:08,518 INFO [Listener at localhost/45551] ipc.NettyRpcServer(120): Bind to /172.31.2.10:42381 2023-02-09 20:54:08,528 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,530 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,550 INFO [Listener at localhost/45551] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42381 connecting to ZooKeeper ensemble=127.0.0.1:58959 2023-02-09 20:54:08,599 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:423810x0, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 20:54:08,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:42381-0x1003addd87e0000 connected 2023-02-09 20:54:08,636 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 20:54:08,637 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:54:08,640 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 20:54:08,649 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42381 2023-02-09 20:54:08,650 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42381 2023-02-09 20:54:08,651 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42381 2023-02-09 20:54:08,653 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42381 2023-02-09 20:54:08,653 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42381 2023-02-09 20:54:08,660 INFO [Listener at localhost/45551] master.HMaster(439): hbase.rootdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686, hbase.cluster.distributed=false 2023-02-09 20:54:08,743 INFO [Listener at localhost/45551] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-09 20:54:08,744 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,744 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,744 INFO [Listener at localhost/45551] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 20:54:08,744 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,744 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 20:54:08,749 INFO [Listener at localhost/45551] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 20:54:08,752 INFO [Listener at localhost/45551] ipc.NettyRpcServer(120): Bind to /172.31.2.10:39321 2023-02-09 20:54:08,754 INFO [Listener at localhost/45551] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 20:54:08,760 DEBUG [Listener at localhost/45551] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 20:54:08,761 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,764 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,766 INFO [Listener at localhost/45551] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39321 connecting to ZooKeeper ensemble=127.0.0.1:58959 2023-02-09 20:54:08,771 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:393210x0, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 20:54:08,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39321-0x1003addd87e0001 connected 2023-02-09 20:54:08,772 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 20:54:08,774 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:54:08,775 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 20:54:08,775 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39321 2023-02-09 20:54:08,776 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39321 2023-02-09 20:54:08,779 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39321 2023-02-09 20:54:08,779 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39321 2023-02-09 20:54:08,780 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39321 2023-02-09 20:54:08,793 INFO [Listener at localhost/45551] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-09 20:54:08,793 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,793 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,794 INFO [Listener at localhost/45551] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 20:54:08,794 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,794 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 20:54:08,795 INFO [Listener at localhost/45551] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 20:54:08,797 INFO [Listener at localhost/45551] ipc.NettyRpcServer(120): Bind to /172.31.2.10:36111 2023-02-09 20:54:08,797 INFO [Listener at localhost/45551] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 20:54:08,798 DEBUG [Listener at localhost/45551] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 20:54:08,799 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,801 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,803 INFO [Listener at localhost/45551] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36111 connecting to ZooKeeper ensemble=127.0.0.1:58959 2023-02-09 20:54:08,809 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:361110x0, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 20:54:08,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36111-0x1003addd87e0002 connected 2023-02-09 20:54:08,810 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 20:54:08,811 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:54:08,812 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 20:54:08,813 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36111 2023-02-09 20:54:08,813 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36111 2023-02-09 20:54:08,814 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36111 2023-02-09 20:54:08,815 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36111 2023-02-09 20:54:08,815 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36111 2023-02-09 20:54:08,828 INFO [Listener at localhost/45551] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-02-09 20:54:08,828 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,829 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,829 INFO [Listener at localhost/45551] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-09 20:54:08,829 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-09 20:54:08,829 INFO [Listener at localhost/45551] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-09 20:54:08,829 INFO [Listener at localhost/45551] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-09 20:54:08,830 INFO [Listener at localhost/45551] ipc.NettyRpcServer(120): Bind to /172.31.2.10:43365 2023-02-09 20:54:08,831 INFO [Listener at localhost/45551] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-09 20:54:08,832 DEBUG [Listener at localhost/45551] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-09 20:54:08,833 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,835 INFO [Listener at localhost/45551] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:08,836 INFO [Listener at localhost/45551] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43365 connecting to ZooKeeper ensemble=127.0.0.1:58959 2023-02-09 20:54:08,840 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:433650x0, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-09 20:54:08,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43365-0x1003addd87e0003 connected 2023-02-09 20:54:08,841 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 20:54:08,842 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:54:08,843 DEBUG [Listener at localhost/45551] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-09 20:54:08,843 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43365 2023-02-09 20:54:08,843 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43365 2023-02-09 20:54:08,844 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43365 2023-02-09 20:54:08,844 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43365 2023-02-09 20:54:08,844 DEBUG [Listener at localhost/45551] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43365 2023-02-09 20:54:08,846 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:08,855 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 20:54:08,856 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:08,875 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 20:54:08,876 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 20:54:08,876 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 20:54:08,875 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-09 20:54:08,876 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:08,877 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 20:54:08,880 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,42381,1675976047566 from backup master directory 2023-02-09 20:54:08,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-09 20:54:08,888 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:08,888 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-09 20:54:08,889 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 20:54:08,889 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:08,891 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-09 20:54:08,893 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-09 20:54:08,984 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/hbase.id with ID: 391620a7-50c2-49f3-ab05-89caa64d6be7 2023-02-09 20:54:09,029 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-09 20:54:09,044 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:09,099 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x240f9ec7 to 127.0.0.1:58959 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 20:54:09,132 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d70c5a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 20:54:09,155 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-09 20:54:09,157 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-09 20:54:09,173 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-09 20:54:09,174 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-09 20:54:09,176 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:54:09,181 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:54:09,183 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 20:54:09,216 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store-tmp 2023-02-09 20:54:09,294 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:09,295 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 20:54:09,295 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:54:09,295 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:54:09,295 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 20:54:09,295 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:54:09,296 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:54:09,296 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675976049295Disabling compacts and flushes for region at 1675976049295Disabling writes for close at 1675976049295Writing region close event to WAL at 1675976049295Closed at 1675976049295 2023-02-09 20:54:09,297 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/WALs/jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:09,317 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C42381%2C1675976047566, suffix=, logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/WALs/jenkins-hbase9.apache.org,42381,1675976047566, archiveDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/oldWALs, maxLogs=10 2023-02-09 20:54:09,371 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK] 2023-02-09 20:54:09,371 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK] 2023-02-09 20:54:09,371 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK] 2023-02-09 20:54:09,380 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:54:09,454 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/WALs/jenkins-hbase9.apache.org,42381,1675976047566/jenkins-hbase9.apache.org%2C42381%2C1675976047566.1675976049327 2023-02-09 20:54:09,454 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK], DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK], DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK]] 2023-02-09 20:54:09,455 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:09,455 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:09,459 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,461 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,529 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,537 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-09 20:54:09,570 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-09 20:54:09,585 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:09,591 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,593 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,611 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-09 20:54:09,616 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:09,617 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69801265, jitterRate=0.040119901299476624}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 20:54:09,617 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1675976049491Initializing all the Stores at 1675976049494 (+3 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976049495 (+1 ms)Cleaning up temporary data from old regions at 1675976049601 (+106 ms)Cleaning up detritus from prior splits at 1675976049602 (+1 ms)Region opened successfully at 1675976049617 (+15 ms) 2023-02-09 20:54:09,619 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-09 20:54:09,644 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:09,644 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:09,647 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:09,649 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-09 20:54:09,690 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 40 msec 2023-02-09 20:54:09,690 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-09 20:54:09,716 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-09 20:54:09,722 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-09 20:54:09,753 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-09 20:54:09,758 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-09 20:54:09,760 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-09 20:54:09,767 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-09 20:54:09,772 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-09 20:54:09,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:09,778 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-09 20:54:09,778 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-09 20:54:09,792 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-09 20:54:09,797 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 20:54:09,797 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 20:54:09,797 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 20:54:09,797 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-09 20:54:09,798 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:09,798 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase9.apache.org,42381,1675976047566, sessionid=0x1003addd87e0000, setting cluster-up flag (Was=false) 2023-02-09 20:54:09,813 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:09,820 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-09 20:54:09,821 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:09,830 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:09,835 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-09 20:54:09,837 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:09,840 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.hbase-snapshot/.tmp 2023-02-09 20:54:09,848 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(952): ClusterId : 391620a7-50c2-49f3-ab05-89caa64d6be7 2023-02-09 20:54:09,849 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(952): ClusterId : 391620a7-50c2-49f3-ab05-89caa64d6be7 2023-02-09 20:54:09,848 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(952): ClusterId : 391620a7-50c2-49f3-ab05-89caa64d6be7 2023-02-09 20:54:09,854 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 20:54:09,854 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 20:54:09,854 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-09 20:54:09,866 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 20:54:09,866 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 20:54:09,866 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-09 20:54:09,867 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 20:54:09,867 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 20:54:09,867 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-09 20:54:09,871 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 20:54:09,871 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 20:54:09,871 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-09 20:54:09,874 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ReadOnlyZKClient(139): Connect 0x41ceeaf1 to 127.0.0.1:58959 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 20:54:09,874 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ReadOnlyZKClient(139): Connect 0x766fd7fc to 127.0.0.1:58959 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 20:54:09,874 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ReadOnlyZKClient(139): Connect 0x79cb1bdf to 127.0.0.1:58959 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 20:54:09,888 DEBUG [RS:0;jenkins-hbase9:39321] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78a83741, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 20:54:09,888 DEBUG [RS:1;jenkins-hbase9:36111] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f090cb8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 20:54:09,889 DEBUG [RS:0;jenkins-hbase9:39321] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c97bc8c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-09 20:54:09,889 DEBUG [RS:1;jenkins-hbase9:36111] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b368181, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-09 20:54:09,889 DEBUG [RS:2;jenkins-hbase9:43365] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@770cea96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 20:54:09,891 DEBUG [RS:2;jenkins-hbase9:43365] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@733370af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-09 20:54:09,912 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:39321 2023-02-09 20:54:09,912 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase9:43365 2023-02-09 20:54:09,916 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase9:36111 2023-02-09 20:54:09,917 INFO [RS:2;jenkins-hbase9:43365] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 20:54:09,917 INFO [RS:1;jenkins-hbase9:36111] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 20:54:09,917 INFO [RS:0;jenkins-hbase9:39321] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-09 20:54:09,918 INFO [RS:0;jenkins-hbase9:39321] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 20:54:09,918 INFO [RS:1;jenkins-hbase9:36111] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 20:54:09,918 INFO [RS:2;jenkins-hbase9:43365] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-09 20:54:09,918 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 20:54:09,918 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 20:54:09,918 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1023): About to register with Master. 2023-02-09 20:54:09,921 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:36111, startcode=1675976048792 2023-02-09 20:54:09,921 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:43365, startcode=1675976048828 2023-02-09 20:54:09,921 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:39321, startcode=1675976048743 2023-02-09 20:54:09,942 DEBUG [RS:2;jenkins-hbase9:43365] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 20:54:09,942 DEBUG [RS:0;jenkins-hbase9:39321] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 20:54:09,942 DEBUG [RS:1;jenkins-hbase9:36111] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-09 20:54:09,991 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-09 20:54:10,002 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,003 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:53233, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 20:54:10,005 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976080005 2023-02-09 20:54:10,003 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:54053, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 20:54:10,003 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:55183, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-09 20:54:10,007 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-09 20:54:10,012 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:10,013 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-09 20:54:10,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42381] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-09 20:54:10,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-09 20:54:10,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-09 20:54:10,019 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 20:54:10,020 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-09 20:54:10,029 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-09 20:54:10,029 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-09 20:54:10,030 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-09 20:54:10,030 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-09 20:54:10,031 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,034 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-09 20:54:10,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-09 20:54:10,036 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-09 20:54:10,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-09 20:54:10,038 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-09 20:54:10,041 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1675976050040,5,FailOnTimeoutGroup] 2023-02-09 20:54:10,042 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1675976050042,5,FailOnTimeoutGroup] 2023-02-09 20:54:10,042 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,042 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-09 20:54:10,044 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,044 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,050 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(2829): Master is not running yet 2023-02-09 20:54:10,051 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(2829): Master is not running yet 2023-02-09 20:54:10,051 WARN [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-09 20:54:10,050 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(2829): Master is not running yet 2023-02-09 20:54:10,051 WARN [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-09 20:54:10,051 WARN [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-09 20:54:10,071 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:10,072 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:10,072 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686 2023-02-09 20:54:10,096 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:10,101 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 20:54:10,104 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/info 2023-02-09 20:54:10,105 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 20:54:10,106 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,106 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 20:54:10,108 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/rep_barrier 2023-02-09 20:54:10,109 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 20:54:10,109 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,110 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 20:54:10,112 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/table 2023-02-09 20:54:10,112 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 20:54:10,113 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,114 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740 2023-02-09 20:54:10,115 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740 2023-02-09 20:54:10,119 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 20:54:10,121 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 20:54:10,124 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:10,125 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70847726, jitterRate=0.055713385343551636}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 20:54:10,125 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1675976050096Initializing all the Stores at 1675976050100 (+4 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675976050101 (+1 ms)Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976050101Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675976050101Cleaning up temporary data from old regions at 1675976050117 (+16 ms)Cleaning up detritus from prior splits at 1675976050117Region opened successfully at 1675976050125 (+8 ms) 2023-02-09 20:54:10,126 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 20:54:10,126 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 20:54:10,126 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 20:54:10,126 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 20:54:10,126 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 20:54:10,127 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 20:54:10,127 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675976050126Disabling compacts and flushes for region at 1675976050126Disabling writes for close at 1675976050126Writing region close event to WAL at 1675976050127 (+1 ms)Closed at 1675976050127 2023-02-09 20:54:10,132 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:10,132 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-09 20:54:10,140 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-09 20:54:10,152 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:39321, startcode=1675976048743 2023-02-09 20:54:10,153 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:36111, startcode=1675976048792 2023-02-09 20:54:10,153 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,42381,1675976047566 with isa=jenkins-hbase9.apache.org/172.31.2.10:43365, startcode=1675976048828 2023-02-09 20:54:10,157 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-09 20:54:10,159 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-09 20:54:10,159 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42381] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,160 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,168 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686 2023-02-09 20:54:10,169 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686 2023-02-09 20:54:10,168 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686 2023-02-09 20:54:10,169 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38591 2023-02-09 20:54:10,169 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38591 2023-02-09 20:54:10,169 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 20:54:10,169 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:38591 2023-02-09 20:54:10,169 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 20:54:10,169 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-09 20:54:10,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:54:10,179 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ZKUtil(162): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,179 WARN [RS:0;jenkins-hbase9:39321] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 20:54:10,179 INFO [RS:0;jenkins-hbase9:39321] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 20:54:10,180 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ZKUtil(162): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,180 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,180 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ZKUtil(162): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,180 WARN [RS:1;jenkins-hbase9:36111] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 20:54:10,180 WARN [RS:2;jenkins-hbase9:43365] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-09 20:54:10,180 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,39321,1675976048743] 2023-02-09 20:54:10,181 INFO [RS:2;jenkins-hbase9:43365] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 20:54:10,180 INFO [RS:1;jenkins-hbase9:36111] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 20:54:10,181 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,181 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1947): logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,181 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,36111,1675976048792] 2023-02-09 20:54:10,182 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,43365,1675976048828] 2023-02-09 20:54:10,193 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ZKUtil(162): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,193 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ZKUtil(162): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,193 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ZKUtil(162): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,194 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ZKUtil(162): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,194 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ZKUtil(162): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,194 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ZKUtil(162): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,194 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ZKUtil(162): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,194 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ZKUtil(162): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,194 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ZKUtil(162): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,204 DEBUG [RS:0;jenkins-hbase9:39321] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 20:54:10,204 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 20:54:10,204 DEBUG [RS:1;jenkins-hbase9:36111] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-09 20:54:10,213 INFO [RS:1;jenkins-hbase9:36111] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 20:54:10,213 INFO [RS:2;jenkins-hbase9:43365] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 20:54:10,213 INFO [RS:0;jenkins-hbase9:39321] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-09 20:54:10,233 INFO [RS:2;jenkins-hbase9:43365] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 20:54:10,233 INFO [RS:1;jenkins-hbase9:36111] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 20:54:10,233 INFO [RS:0;jenkins-hbase9:39321] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-09 20:54:10,237 INFO [RS:1;jenkins-hbase9:36111] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 20:54:10,237 INFO [RS:2;jenkins-hbase9:43365] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 20:54:10,238 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,237 INFO [RS:0;jenkins-hbase9:39321] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-09 20:54:10,238 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,238 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,238 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 20:54:10,239 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 20:54:10,239 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-09 20:54:10,246 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,246 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,246 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,246 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,246 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-09 20:54:10,247 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,247 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:0;jenkins-hbase9:39321] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,248 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-09 20:54:10,249 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-02-09 20:54:10,249 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:2;jenkins-hbase9:43365] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,249 DEBUG [RS:1;jenkins-hbase9:36111] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-02-09 20:54:10,251 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,251 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,251 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,254 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,255 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,255 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,260 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,260 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,261 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,271 INFO [RS:1;jenkins-hbase9:36111] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 20:54:10,271 INFO [RS:0;jenkins-hbase9:39321] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 20:54:10,272 INFO [RS:2;jenkins-hbase9:43365] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-09 20:54:10,274 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,39321,1675976048743-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,274 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,36111,1675976048792-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,274 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,43365,1675976048828-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,297 INFO [RS:2;jenkins-hbase9:43365] regionserver.Replication(203): jenkins-hbase9.apache.org,43365,1675976048828 started 2023-02-09 20:54:10,297 INFO [RS:0;jenkins-hbase9:39321] regionserver.Replication(203): jenkins-hbase9.apache.org,39321,1675976048743 started 2023-02-09 20:54:10,298 INFO [RS:1;jenkins-hbase9:36111] regionserver.Replication(203): jenkins-hbase9.apache.org,36111,1675976048792 started 2023-02-09 20:54:10,298 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,39321,1675976048743, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:39321, sessionid=0x1003addd87e0001 2023-02-09 20:54:10,297 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,43365,1675976048828, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:43365, sessionid=0x1003addd87e0003 2023-02-09 20:54:10,298 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,36111,1675976048792, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:36111, sessionid=0x1003addd87e0002 2023-02-09 20:54:10,298 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 20:54:10,298 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 20:54:10,298 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-09 20:54:10,298 DEBUG [RS:1;jenkins-hbase9:36111] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,298 DEBUG [RS:0;jenkins-hbase9:39321] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,299 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,36111,1675976048792' 2023-02-09 20:54:10,299 DEBUG [RS:2;jenkins-hbase9:43365] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,300 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:10,299 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,39321,1675976048743' 2023-02-09 20:54:10,300 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:10,300 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,43365,1675976048828' 2023-02-09 20:54:10,300 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:10,300 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:10,300 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:10,300 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:10,301 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 20:54:10,301 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 20:54:10,301 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 20:54:10,301 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-09 20:54:10,301 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 20:54:10,301 DEBUG [RS:0;jenkins-hbase9:39321] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:10,301 DEBUG [RS:1;jenkins-hbase9:36111] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:10,301 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-09 20:54:10,302 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,36111,1675976048792' 2023-02-09 20:54:10,302 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,39321,1675976048743' 2023-02-09 20:54:10,302 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 20:54:10,302 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 20:54:10,302 DEBUG [RS:2;jenkins-hbase9:43365] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,302 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,43365,1675976048828' 2023-02-09 20:54:10,302 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-09 20:54:10,303 DEBUG [RS:0;jenkins-hbase9:39321] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 20:54:10,303 DEBUG [RS:1;jenkins-hbase9:36111] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 20:54:10,303 DEBUG [RS:2;jenkins-hbase9:43365] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-09 20:54:10,303 DEBUG [RS:0;jenkins-hbase9:39321] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 20:54:10,303 DEBUG [RS:1;jenkins-hbase9:36111] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 20:54:10,303 INFO [RS:0;jenkins-hbase9:39321] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 20:54:10,303 INFO [RS:1;jenkins-hbase9:36111] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 20:54:10,303 DEBUG [RS:2;jenkins-hbase9:43365] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-09 20:54:10,305 INFO [RS:1;jenkins-hbase9:36111] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 20:54:10,305 INFO [RS:0;jenkins-hbase9:39321] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 20:54:10,305 INFO [RS:2;jenkins-hbase9:43365] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-09 20:54:10,305 INFO [RS:2;jenkins-hbase9:43365] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-09 20:54:10,311 DEBUG [jenkins-hbase9:42381] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-09 20:54:10,314 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:10,320 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:10,320 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:10,320 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:10,320 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:10,322 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,43365,1675976048828, state=OPENING 2023-02-09 20:54:10,329 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-09 20:54:10,330 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:10,331 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 20:54:10,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:10,416 INFO [RS:1;jenkins-hbase9:36111] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C36111%2C1675976048792, suffix=, logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,36111,1675976048792, archiveDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs, maxLogs=32 2023-02-09 20:54:10,416 INFO [RS:0;jenkins-hbase9:39321] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C39321%2C1675976048743, suffix=, logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,39321,1675976048743, archiveDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs, maxLogs=32 2023-02-09 20:54:10,416 INFO [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C43365%2C1675976048828, suffix=, logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,43365,1675976048828, archiveDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs, maxLogs=32 2023-02-09 20:54:10,443 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK] 2023-02-09 20:54:10,444 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK] 2023-02-09 20:54:10,443 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK] 2023-02-09 20:54:10,444 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK] 2023-02-09 20:54:10,444 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK] 2023-02-09 20:54:10,444 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK] 2023-02-09 20:54:10,446 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK] 2023-02-09 20:54:10,450 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK] 2023-02-09 20:54:10,450 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK] 2023-02-09 20:54:10,461 INFO [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,43365,1675976048828/jenkins-hbase9.apache.org%2C43365%2C1675976048828.1675976050420 2023-02-09 20:54:10,461 INFO [RS:1;jenkins-hbase9:36111] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,36111,1675976048792/jenkins-hbase9.apache.org%2C36111%2C1675976048792.1675976050421 2023-02-09 20:54:10,463 DEBUG [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK], DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK], DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK]] 2023-02-09 20:54:10,463 INFO [RS:0;jenkins-hbase9:39321] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,39321,1675976048743/jenkins-hbase9.apache.org%2C39321%2C1675976048743.1675976050420 2023-02-09 20:54:10,464 DEBUG [RS:1;jenkins-hbase9:36111] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK], DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK], DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK]] 2023-02-09 20:54:10,465 DEBUG [RS:0;jenkins-hbase9:39321] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK], DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK], DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK]] 2023-02-09 20:54:10,519 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:10,522 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 20:54:10,525 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 20:54:10,538 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-09 20:54:10,539 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-09 20:54:10,543 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C43365%2C1675976048828.meta, suffix=.meta, logDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,43365,1675976048828, archiveDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs, maxLogs=32 2023-02-09 20:54:10,574 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK] 2023-02-09 20:54:10,577 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK] 2023-02-09 20:54:10,577 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK] 2023-02-09 20:54:10,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/WALs/jenkins-hbase9.apache.org,43365,1675976048828/jenkins-hbase9.apache.org%2C43365%2C1675976048828.meta.1675976050544.meta 2023-02-09 20:54:10,582 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41149,DS-d0fad12c-c922-4412-b45d-76cca16cdd0d,DISK], DatanodeInfoWithStorage[127.0.0.1:40557,DS-36ecabf6-9ef8-4440-8798-1ec372920440,DISK], DatanodeInfoWithStorage[127.0.0.1:44401,DS-50604c3e-2bff-4d01-b8e9-1956e6dc7002,DISK]] 2023-02-09 20:54:10,583 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:10,591 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:10,592 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-09 20:54:10,614 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-09 20:54:10,617 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-09 20:54:10,622 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-09 20:54:10,622 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:10,622 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-09 20:54:10,622 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-09 20:54:10,625 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-09 20:54:10,627 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/info 2023-02-09 20:54:10,627 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/info 2023-02-09 20:54:10,628 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-09 20:54:10,629 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,629 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-09 20:54:10,631 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/rep_barrier 2023-02-09 20:54:10,631 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/rep_barrier 2023-02-09 20:54:10,631 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-09 20:54:10,632 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,632 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-09 20:54:10,634 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/table 2023-02-09 20:54:10,634 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/table 2023-02-09 20:54:10,635 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-09 20:54:10,636 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:10,639 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740 2023-02-09 20:54:10,645 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740 2023-02-09 20:54:10,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-09 20:54:10,653 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-09 20:54:10,655 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74567388, jitterRate=0.1111406683921814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-09 20:54:10,655 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1675976050622Writing region info on filesystem at 1675976050623 (+1 ms)Initializing all the Stores at 1675976050625 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675976050625Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976050625Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675976050625Cleaning up temporary data from old regions at 1675976050648 (+23 ms)Cleaning up detritus from prior splits at 1675976050648Running coprocessor post-open hooks at 1675976050655 (+7 ms)Region opened successfully at 1675976050655 2023-02-09 20:54:10,667 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1675976050511 2023-02-09 20:54:10,691 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-09 20:54:10,697 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-09 20:54:10,700 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,43365,1675976048828, state=OPEN 2023-02-09 20:54:10,703 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-09 20:54:10,703 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-09 20:54:10,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-09 20:54:10,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,43365,1675976048828 in 369 msec 2023-02-09 20:54:10,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-09 20:54:10,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 569 msec 2023-02-09 20:54:10,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 817 msec 2023-02-09 20:54:10,723 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1675976048871, completionTime=-1 2023-02-09 20:54:10,724 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-09 20:54:10,724 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:10,797 DEBUG [hconnection-0x3e91e44f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 20:54:10,804 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 20:54:10,824 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:10,825 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976110825 2023-02-09 20:54:10,825 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976170825 2023-02-09 20:54:10,825 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 100 msec 2023-02-09 20:54:10,851 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,42381,1675976047566-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,852 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,42381,1675976047566-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,852 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,42381,1675976047566-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,853 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:42381, period=300000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,854 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-09 20:54:10,861 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-09 20:54:10,869 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-09 20:54:10,871 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-09 20:54:10,881 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:10,884 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:10,886 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:10,910 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:10,912 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596 empty. 2023-02-09 20:54:10,913 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:10,913 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-09 20:54:10,949 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:10,951 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9a9aa832e601f0693eaa144f8209e596, NAME => 'hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:10,970 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:10,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 9a9aa832e601f0693eaa144f8209e596, disabling compactions & flushes 2023-02-09 20:54:10,971 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:10,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:10,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. after waiting 0 ms 2023-02-09 20:54:10,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:10,971 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:10,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 9a9aa832e601f0693eaa144f8209e596: Waiting for close lock at 1675976050971Disabling compacts and flushes for region at 1675976050971Disabling writes for close at 1675976050971Writing region close event to WAL at 1675976050971Closed at 1675976050971 2023-02-09 20:54:10,976 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:10,990 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675976050978"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976050978"}]},"ts":"1675976050978"} 2023-02-09 20:54:11,019 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:11,021 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:11,026 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976051021"}]},"ts":"1675976051021"} 2023-02-09 20:54:11,030 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-09 20:54:11,035 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:11,036 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:11,036 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:11,036 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:11,036 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:11,039 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9a9aa832e601f0693eaa144f8209e596, ASSIGN}] 2023-02-09 20:54:11,042 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=9a9aa832e601f0693eaa144f8209e596, ASSIGN 2023-02-09 20:54:11,043 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=9a9aa832e601f0693eaa144f8209e596, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:54:11,196 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:11,197 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9a9aa832e601f0693eaa144f8209e596, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:11,197 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675976051197"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976051197"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976051197"}]},"ts":"1675976051197"} 2023-02-09 20:54:11,202 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 9a9aa832e601f0693eaa144f8209e596, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:11,362 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:11,362 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9a9aa832e601f0693eaa144f8209e596, NAME => 'hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:11,364 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:11,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:11,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,366 INFO [StoreOpener-9a9aa832e601f0693eaa144f8209e596-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,368 DEBUG [StoreOpener-9a9aa832e601f0693eaa144f8209e596-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/info 2023-02-09 20:54:11,369 DEBUG [StoreOpener-9a9aa832e601f0693eaa144f8209e596-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/info 2023-02-09 20:54:11,369 INFO [StoreOpener-9a9aa832e601f0693eaa144f8209e596-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a9aa832e601f0693eaa144f8209e596 columnFamilyName info 2023-02-09 20:54:11,370 INFO [StoreOpener-9a9aa832e601f0693eaa144f8209e596-1] regionserver.HStore(310): Store=9a9aa832e601f0693eaa144f8209e596/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:11,372 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,372 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,376 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:54:11,380 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:11,380 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 9a9aa832e601f0693eaa144f8209e596; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71982927, jitterRate=0.07262919843196869}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 20:54:11,381 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 9a9aa832e601f0693eaa144f8209e596: Running coprocessor pre-open hook at 1675976051365Writing region info on filesystem at 1675976051365Initializing all the Stores at 1675976051366 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1675976051366Cleaning up temporary data from old regions at 1675976051373 (+7 ms)Cleaning up detritus from prior splits at 1675976051374 (+1 ms)Running coprocessor post-open hooks at 1675976051381 (+7 ms)Region opened successfully at 1675976051381 2023-02-09 20:54:11,383 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596., pid=6, masterSystemTime=1675976051356 2023-02-09 20:54:11,386 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:11,386 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:54:11,388 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:11,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1675976051387"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976051387"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976051387"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976051387"}]},"ts":"1675976051387"} 2023-02-09 20:54:11,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-09 20:54:11,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 9a9aa832e601f0693eaa144f8209e596, server=jenkins-hbase9.apache.org,43365,1675976048828 in 191 msec 2023-02-09 20:54:11,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-09 20:54:11,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=9a9aa832e601f0693eaa144f8209e596, ASSIGN in 359 msec 2023-02-09 20:54:11,402 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:11,403 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976051402"}]},"ts":"1675976051402"} 2023-02-09 20:54:11,405 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-09 20:54:11,409 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:11,411 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 537 msec 2023-02-09 20:54:11,484 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-09 20:54:11,486 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-09 20:54:11,486 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:11,520 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:11,537 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 20:54:11,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-02-09 20:54:11,553 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:11,566 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-09 20:54:11,571 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-09 20:54:11,579 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-09 20:54:11,583 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-09 20:54:11,583 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.694sec 2023-02-09 20:54:11,587 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-09 20:54:11,588 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-09 20:54:11,588 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-09 20:54:11,590 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,42381,1675976047566-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-09 20:54:11,592 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,42381,1675976047566-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-09 20:54:11,600 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-09 20:54:11,654 DEBUG [Listener at localhost/45551] zookeeper.ReadOnlyZKClient(139): Connect 0x37e887de to 127.0.0.1:58959 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-09 20:54:11,659 DEBUG [Listener at localhost/45551] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c8d5aec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-09 20:54:11,673 DEBUG [hconnection-0x7e2f5f25-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 20:54:11,686 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 20:54:11,696 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:54:11,723 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=358, OpenFileDescriptor=693, MaxFileDescriptor=60000, SystemLoadAverage=278, ProcessCount=168, AvailableMemoryMB=11036 2023-02-09 20:54:11,729 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:11,738 DEBUG [Listener at localhost/45551] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-09 20:54:11,742 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:55528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-09 20:54:11,757 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:11,757 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:54:11,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:11,764 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:11,766 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:11,768 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:11,771 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:11,772 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 empty. 2023-02-09 20:54:11,774 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:11,774 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-02-09 20:54:11,797 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:11,799 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => ec5090c9a8a7e77ccbbcbcb4ed761099, NAME => 'testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionUnevenDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1603): Closing ec5090c9a8a7e77ccbbcbcb4ed761099, disabling compactions & flushes 2023-02-09 20:54:11,850 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. after waiting 0 ms 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:11,850 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:11,850 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1557): Region close journal for ec5090c9a8a7e77ccbbcbcb4ed761099: Waiting for close lock at 1675976051850Disabling compacts and flushes for region at 1675976051850Disabling writes for close at 1675976051850Writing region close event to WAL at 1675976051850Closed at 1675976051850 2023-02-09 20:54:11,854 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:11,856 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976051856"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976051856"}]},"ts":"1675976051856"} 2023-02-09 20:54:11,858 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:11,860 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:11,860 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976051860"}]},"ts":"1675976051860"} 2023-02-09 20:54:11,862 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLING in hbase:meta 2023-02-09 20:54:11,866 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:11,867 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:11,867 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:11,868 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:11,868 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:11,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, ASSIGN}] 2023-02-09 20:54:11,870 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, ASSIGN 2023-02-09 20:54:11,872 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:12,022 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:12,023 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ec5090c9a8a7e77ccbbcbcb4ed761099, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,023 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976052023"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976052023"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976052023"}]},"ts":"1675976052023"} 2023-02-09 20:54:12,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure ec5090c9a8a7e77ccbbcbcb4ed761099, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:12,181 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,181 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 20:54:12,184 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:50952, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 20:54:12,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ec5090c9a8a7e77ccbbcbcb4ed761099, NAME => 'testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:12,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:12,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:12,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,193 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,196 DEBUG [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1 2023-02-09 20:54:12,196 DEBUG [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1 2023-02-09 20:54:12,196 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec5090c9a8a7e77ccbbcbcb4ed761099 columnFamilyName cf1 2023-02-09 20:54:12,197 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] regionserver.HStore(310): Store=ec5090c9a8a7e77ccbbcbcb4ed761099/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:12,197 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,199 DEBUG [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2 2023-02-09 20:54:12,199 DEBUG [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2 2023-02-09 20:54:12,199 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ec5090c9a8a7e77ccbbcbcb4ed761099 columnFamilyName cf2 2023-02-09 20:54:12,200 INFO [StoreOpener-ec5090c9a8a7e77ccbbcbcb4ed761099-1] regionserver.HStore(310): Store=ec5090c9a8a7e77ccbbcbcb4ed761099/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:12,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:12,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:12,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:12,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened ec5090c9a8a7e77ccbbcbcb4ed761099; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63882555, jitterRate=-0.04807575047016144}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:12,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for ec5090c9a8a7e77ccbbcbcb4ed761099: Running coprocessor pre-open hook at 1675976052192Writing region info on filesystem at 1675976052192Initializing all the Stores at 1675976052193 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976052193Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976052193Cleaning up temporary data from old regions at 1675976052204 (+11 ms)Cleaning up detritus from prior splits at 1675976052204Running coprocessor post-open hooks at 1675976052211 (+7 ms)Region opened successfully at 1675976052211 2023-02-09 20:54:12,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099., pid=11, masterSystemTime=1675976052181 2023-02-09 20:54:12,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,219 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=ec5090c9a8a7e77ccbbcbcb4ed761099, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,219 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976052218"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976052218"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976052218"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976052218"}]},"ts":"1675976052218"} 2023-02-09 20:54:12,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-02-09 20:54:12,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure ec5090c9a8a7e77ccbbcbcb4ed761099, server=jenkins-hbase9.apache.org,36111,1675976048792 in 196 msec 2023-02-09 20:54:12,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-02-09 20:54:12,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, ASSIGN in 358 msec 2023-02-09 20:54:12,230 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:12,230 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976052230"}]},"ts":"1675976052230"} 2023-02-09 20:54:12,232 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLED in hbase:meta 2023-02-09 20:54:12,235 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:12,238 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter in 474 msec 2023-02-09 20:54:12,269 DEBUG [Listener at localhost/45551] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 20:54:12,272 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:50954, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 20:54:12,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:12,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,300 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,300 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,301 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:12,301 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:12,301 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,301 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,303 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,303 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,303 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,303 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,304 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,304 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,309 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,309 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,309 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,309 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,308 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,309 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:12,310 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:12,310 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:12,311 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,382 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/c16d4e70eed845099ea7e34073532c8d 2023-02-09 20:54:12,441 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f 2023-02-09 20:54:12,450 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/c16d4e70eed845099ea7e34073532c8d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d 2023-02-09 20:54:12,461 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:12,463 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f 2023-02-09 20:54:12,473 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:12,475 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 164ms, sequenceid=6, compaction requested=false 2023-02-09 20:54:12,476 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,478 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:12,478 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,478 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:12,479 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:12,479 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,479 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,479 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,481 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,481 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,482 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,482 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:12,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:12,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db29a50[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:12,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:12,491 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,492 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:12,492 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:12,494 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,494 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,494 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,495 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:12,500 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:12,500 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:12,500 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:12,501 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,501 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,505 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,506 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,506 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,509 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,509 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,509 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,521 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:12,522 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,522 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,524 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,524 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,525 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:12,525 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:12,625 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:12,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:12,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:12,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:12,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,657 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,657 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,658 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:12,658 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:12,659 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,659 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,660 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,660 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,660 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,660 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,661 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,661 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,664 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,663 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,665 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:12,665 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:12,665 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,666 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,668 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,688 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 2023-02-09 20:54:12,716 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/e49c7332f8d747db94e3aff1ebb1176e 2023-02-09 20:54:12,724 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 2023-02-09 20:54:12,732 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:12,733 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/e49c7332f8d747db94e3aff1ebb1176e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e 2023-02-09 20:54:12,741 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:12,742 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 76ms, sequenceid=19, compaction requested=false 2023-02-09 20:54:12,742 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:12,742 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,742 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:12,744 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:12,744 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,744 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,744 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,746 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,746 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,747 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,747 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:12,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca87e33[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:12,750 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:12,750 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,752 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,752 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:12,752 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,752 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,753 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:12,752 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,753 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,753 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,753 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,755 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,755 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,756 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:12,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:12,761 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:12,761 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:12,761 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:12,761 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,761 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:12,763 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,763 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,763 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,765 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,766 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,766 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,769 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,769 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,770 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,780 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,781 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:12,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:12,781 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,782 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,783 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:12,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:12,883 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:12,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:12,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:12,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,909 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:12,909 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:12,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:12,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,911 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,912 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,912 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,912 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,913 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,913 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,914 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:12,914 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,914 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,915 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,914 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,915 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:12,915 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,916 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,916 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:12,916 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:12,916 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:12,917 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:12,917 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:12,917 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,917 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:12,917 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:12,918 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,919 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,919 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,919 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,920 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,920 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,936 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/7753f0813ed847adaee39a2e8cf2afcc 2023-02-09 20:54:12,959 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/1d7dc89f423c4e8b813ae100f5dc6839 2023-02-09 20:54:12,968 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/7753f0813ed847adaee39a2e8cf2afcc as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc 2023-02-09 20:54:12,975 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:12,976 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/1d7dc89f423c4e8b813ae100f5dc6839 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839 2023-02-09 20:54:12,984 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:12,986 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 69ms, sequenceid=32, compaction requested=true 2023-02-09 20:54:12,986 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:12,986 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:12,986 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:12,987 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:12,987 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,987 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:12,987 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:12,990 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,990 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,991 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,991 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:12,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:12,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760a9ac7[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:12,993 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:12,993 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,994 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,994 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,994 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:12,995 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:12,995 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:12,997 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,997 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,997 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:12,997 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,998 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,998 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,998 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:12,999 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:12,998 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:12,999 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:13,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:13,003 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:13,003 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:13,003 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:13,003 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,003 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,006 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,007 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,007 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,008 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,010 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,011 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,011 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,016 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,017 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:13,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,017 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,018 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,118 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:13,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:13,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:13,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:13,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,143 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,143 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,144 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:13,144 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:13,144 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,144 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,146 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,146 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,147 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,147 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,149 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,149 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,149 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,150 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,151 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,151 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,151 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,152 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:13,152 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:13,152 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,152 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:13,152 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:13,153 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,155 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,155 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,174 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/0d94c953d302489690d45a22a3b65560 2023-02-09 20:54:13,198 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/72a95a7e95df448fbcf0577870d9eba0 2023-02-09 20:54:13,206 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/0d94c953d302489690d45a22a3b65560 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560 2023-02-09 20:54:13,212 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:13,213 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/72a95a7e95df448fbcf0577870d9eba0 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0 2023-02-09 20:54:13,221 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:13,223 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 71ms, sequenceid=45, compaction requested=true 2023-02-09 20:54:13,223 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:13,223 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,223 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:13,224 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:13,224 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,224 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,224 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,229 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@443b9fe6[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:13,232 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:13,232 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,233 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,233 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,233 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:13,233 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:13,234 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:13,236 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,236 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,236 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,238 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,238 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,238 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,238 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,239 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,239 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,239 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:13,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:13,243 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:13,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:13,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:13,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,243 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,245 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,246 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,246 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,248 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,249 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,249 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,254 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,254 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,255 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:13,255 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,255 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,255 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,256 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:13,356 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:13,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:13,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:13,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:13,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,382 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,383 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,383 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:13,383 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:13,384 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,384 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,385 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,385 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,385 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,385 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,386 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,386 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,387 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,387 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,388 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,388 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,391 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,390 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,391 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,391 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,391 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,393 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,393 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,391 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,393 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,393 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:13,393 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:13,393 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,393 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:13,393 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:13,394 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,394 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,395 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,395 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,395 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,395 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,411 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/fb081348553d4ed899daff38f1a5ff6e 2023-02-09 20:54:13,437 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/0a72afd3e738431baa01ed8764e5eba9 2023-02-09 20:54:13,444 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/fb081348553d4ed899daff38f1a5ff6e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e 2023-02-09 20:54:13,451 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:13,452 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/0a72afd3e738431baa01ed8764e5eba9 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9 2023-02-09 20:54:13,459 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:13,460 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 67ms, sequenceid=58, compaction requested=true 2023-02-09 20:54:13,460 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:13,460 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:13,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:13,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,462 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,462 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,463 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,463 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,464 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,464 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:13,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da2fd2b[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:13,466 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:13,466 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,469 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,469 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:13,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,469 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:13,471 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,471 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,471 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,471 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,472 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,472 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,472 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,472 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,472 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,473 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:13,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:13,478 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:13,478 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:13,478 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:13,478 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,478 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,481 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,481 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,482 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,490 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:13,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,490 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,491 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,592 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:13,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:13,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:13,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:13,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,617 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,617 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,617 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-02-09 20:54:13,618 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-02-09 20:54:13,618 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,618 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,619 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,619 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,619 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,619 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,620 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,620 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,621 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,621 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,621 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,622 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-02-09 20:54:13,622 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,622 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:13,623 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:13,623 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. started... 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,623 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,624 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:13,625 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,625 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,625 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,625 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,625 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,626 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,641 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/cf2985c841e14522a8fc37296f348881 2023-02-09 20:54:13,660 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c 2023-02-09 20:54:13,668 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/cf2985c841e14522a8fc37296f348881 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881 2023-02-09 20:54:13,675 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:13,677 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c 2023-02-09 20:54:13,684 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:13,685 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 61ms, sequenceid=71, compaction requested=true 2023-02-09 20:54:13,685 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ec5090c9a8a7e77ccbbcbcb4ed761099: 2023-02-09 20:54:13,686 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:13,686 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:13,687 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:13,687 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,687 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-02-09 20:54:13,687 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-02-09 20:54:13,689 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-02-09 20:54:13,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12d4454a[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:13,692 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:13,692 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,695 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:13,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,695 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,696 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-02-09 20:54:13,696 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,696 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-02-09 20:54:13,696 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:13,696 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:13,698 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,698 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,698 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,699 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,699 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-02-09 20:54:13,699 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,699 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,700 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,700 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,701 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-02-09 20:54:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:13,705 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-02-09 20:54:13,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:13,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:13,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,705 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:13,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:13,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,708 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,709 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,709 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,711 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,712 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,712 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,715 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:13,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:13,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-02-09 20:54:13,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:13,716 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:13,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,716 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-02-09 20:54:13,718 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:13,818 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:13,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:13,844 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,851 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:13,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:13,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:13,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:13,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, UNASSIGN}] 2023-02-09 20:54:13,861 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, UNASSIGN 2023-02-09 20:54:13,862 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ec5090c9a8a7e77ccbbcbcb4ed761099, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:13,862 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976053862"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976053862"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976053862"}]},"ts":"1675976053862"} 2023-02-09 20:54:13,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure ec5090c9a8a7e77ccbbcbcb4ed761099, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:14,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing ec5090c9a8a7e77ccbbcbcb4ed761099, disabling compactions & flushes 2023-02-09 20:54:14,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:14,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:14,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. after waiting 0 ms 2023-02-09 20:54:14,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:14,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing ec5090c9a8a7e77ccbbcbcb4ed761099 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:54:14,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/570ad7f97d7447108a891e74a3532db5 2023-02-09 20:54:14,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/97bae105c5f04274974ed6e31c13a977 2023-02-09 20:54:14,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf1/570ad7f97d7447108a891e74a3532db5 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5 2023-02-09 20:54:14,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:14,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.tmp/cf2/97bae105c5f04274974ed6e31c13a977 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977 2023-02-09 20:54:14,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:14,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 63ms, sequenceid=82, compaction requested=true 2023-02-09 20:54:14,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:54:14,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:14,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. 2023-02-09 20:54:14,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for ec5090c9a8a7e77ccbbcbcb4ed761099: Waiting for close lock at 1675976054025Running coprocessor pre-close hooks at 1675976054025Disabling compacts and flushes for region at 1675976054025Disabling writes for close at 1675976054025Obtaining lock to block concurrent updates at 1675976054025Preparing flush snapshotting stores in ec5090c9a8a7e77ccbbcbcb4ed761099 at 1675976054025Finished memstore snapshotting testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976054025Flushing stores of testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. at 1675976054026 (+1 ms)Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf1: creating writer at 1675976054026Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf1: appending metadata at 1675976054030 (+4 ms)Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf1: closing flushed file at 1675976054030Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf2: creating writer at 1675976054050 (+20 ms)Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf2: appending metadata at 1675976054052 (+2 ms)Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf2: closing flushed file at 1675976054052Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf1: reopening flushed file at 1675976054072 (+20 ms)Flushing ec5090c9a8a7e77ccbbcbcb4ed761099/cf2: reopening flushed file at 1675976054082 (+10 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for ec5090c9a8a7e77ccbbcbcb4ed761099 in 63ms, sequenceid=82, compaction requested=true at 1675976054088 (+6 ms)Writing region close event to WAL at 1675976054104 (+16 ms)Running coprocessor post-close hooks at 1675976054109 (+5 ms)Closed at 1675976054109 2023-02-09 20:54:14,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=ec5090c9a8a7e77ccbbcbcb4ed761099, regionState=CLOSED 2023-02-09 20:54:14,113 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976054113"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976054113"}]},"ts":"1675976054113"} 2023-02-09 20:54:14,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-02-09 20:54:14,118 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure ec5090c9a8a7e77ccbbcbcb4ed761099, server=jenkins-hbase9.apache.org,36111,1675976048792 in 251 msec 2023-02-09 20:54:14,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-02-09 20:54:14,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ec5090c9a8a7e77ccbbcbcb4ed761099, UNASSIGN in 258 msec 2023-02-09 20:54:14,139 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=12 splitting 14 storefiles, region=ec5090c9a8a7e77ccbbcbcb4ed761099, threads=14 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,140 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,141 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,141 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,141 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,141 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,141 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,184 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9, top=true 2023-02-09 20:54:14,191 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977, top=true 2023-02-09 20:54:14,194 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5, top=true 2023-02-09 20:54:14,214 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c, top=true 2023-02-09 20:54:14,215 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e, top=true 2023-02-09 20:54:14,217 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881, top=true 2023-02-09 20:54:14,284 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9 for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,284 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,311 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5 for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,311 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,324 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977 for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,324 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,325 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,331 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,324 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,331 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,326 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/.splits/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881 for child: 11c5f86a4b18d67a5f0febb048d48b9f, parent: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,331 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,355 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,370 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,379 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,386 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,398 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,400 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,741 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,741 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 for region: ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:14,741 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region ec5090c9a8a7e77ccbbcbcb4ed761099 Daughter A: 8 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:14,794 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:14,796 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:14,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1675976054798"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1675976054798"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1675976054798"}]},"ts":"1675976054798"} 2023-02-09 20:54:14,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976054798"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976054798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976054798"}]},"ts":"1675976054798"} 2023-02-09 20:54:14,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976054798"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976054798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976054798"}]},"ts":"1675976054798"} 2023-02-09 20:54:14,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, ASSIGN}] 2023-02-09 20:54:14,851 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, ASSIGN 2023-02-09 20:54:14,852 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:14,853 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, ASSIGN 2023-02-09 20:54:14,854 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:15,002 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:54:15,004 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=11c5f86a4b18d67a5f0febb048d48b9f, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,004 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055004"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976055004"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976055004"}]},"ts":"1675976055004"} 2023-02-09 20:54:15,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure 11c5f86a4b18d67a5f0febb048d48b9f, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:15,010 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=81b4f32bb42c8db9e70dbb1fb15261b9, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,010 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055010"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976055010"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976055010"}]},"ts":"1675976055010"} 2023-02-09 20:54:15,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure 81b4f32bb42c8db9e70dbb1fb15261b9, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:15,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 81b4f32bb42c8db9e70dbb1fb15261b9, NAME => 'testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.', STARTKEY => '', ENDKEY => '26'} 2023-02-09 20:54:15,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:15,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:15,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,166 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,167 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1 2023-02-09 20:54:15,168 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1 2023-02-09 20:54:15,168 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81b4f32bb42c8db9e70dbb1fb15261b9 columnFamilyName cf1 2023-02-09 20:54:15,202 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-bottom 2023-02-09 20:54:15,210 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-bottom 2023-02-09 20:54:15,216 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-bottom 2023-02-09 20:54:15,223 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-bottom 2023-02-09 20:54:15,224 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(310): Store=81b4f32bb42c8db9e70dbb1fb15261b9/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:15,224 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,225 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2 2023-02-09 20:54:15,225 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2 2023-02-09 20:54:15,226 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81b4f32bb42c8db9e70dbb1fb15261b9 columnFamilyName cf2 2023-02-09 20:54:15,242 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-bottom 2023-02-09 20:54:15,249 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-bottom 2023-02-09 20:54:15,256 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-bottom 2023-02-09 20:54:15,263 DEBUG [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-bottom 2023-02-09 20:54:15,264 INFO [StoreOpener-81b4f32bb42c8db9e70dbb1fb15261b9-1] regionserver.HStore(310): Store=81b4f32bb42c8db9e70dbb1fb15261b9/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:15,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:15,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 81b4f32bb42c8db9e70dbb1fb15261b9; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62672785, jitterRate=-0.06610272824764252}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:15,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 81b4f32bb42c8db9e70dbb1fb15261b9: Running coprocessor pre-open hook at 1675976055165Writing region info on filesystem at 1675976055165Initializing all the Stores at 1675976055166 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976055166Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976055166Cleaning up temporary data from old regions at 1675976055268 (+102 ms)Cleaning up detritus from prior splits at 1675976055268Running coprocessor post-open hooks at 1675976055272 (+4 ms)Region opened successfully at 1675976055272 2023-02-09 20:54:15,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9., pid=18, masterSystemTime=1675976055159 2023-02-09 20:54:15,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:15,277 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-02-09 20:54:15,277 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-02-09 20:54:15,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,280 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,280 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1912): 81b4f32bb42c8db9e70dbb1fb15261b9/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:15,280 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 81b4f32bb42c8db9e70dbb1fb15261b9/cf1 in testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,280 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,281 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1912): 81b4f32bb42c8db9e70dbb1fb15261b9/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:15,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,281 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 81b4f32bb42c8db9e70dbb1fb15261b9/cf2 in testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,281 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=81b4f32bb42c8db9e70dbb1fb15261b9, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11c5f86a4b18d67a5f0febb048d48b9f, NAME => 'testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.', STARTKEY => '26', ENDKEY => ''} 2023-02-09 20:54:15,281 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/.tmp, totalSize=19.9 K 2023-02-09 20:54:15,281 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055281"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976055281"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976055281"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976055281"}]},"ts":"1675976055281"} 2023-02-09 20:54:15,281 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/.tmp, totalSize=19.9 K 2023-02-09 20:54:15,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:15,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:15,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,284 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976052273 2023-02-09 20:54:15,285 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976052273 2023-02-09 20:54:15,286 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976052633 2023-02-09 20:54:15,286 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976052633 2023-02-09 20:54:15,286 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,287 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976052885 2023-02-09 20:54:15,287 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976052885 2023-02-09 20:54:15,287 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976053121 2023-02-09 20:54:15,288 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976053121 2023-02-09 20:54:15,288 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1 2023-02-09 20:54:15,288 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1 2023-02-09 20:54:15,289 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11c5f86a4b18d67a5f0febb048d48b9f columnFamilyName cf1 2023-02-09 20:54:15,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-02-09 20:54:15,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure 81b4f32bb42c8db9e70dbb1fb15261b9, server=jenkins-hbase9.apache.org,36111,1675976048792 in 274 msec 2023-02-09 20:54:15,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, ASSIGN in 442 msec 2023-02-09 20:54:15,308 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): 81b4f32bb42c8db9e70dbb1fb15261b9#cf2#compaction#14 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:15,309 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-top 2023-02-09 20:54:15,313 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): 81b4f32bb42c8db9e70dbb1fb15261b9#cf1#compaction#15 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:15,315 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-top 2023-02-09 20:54:15,332 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-top 2023-02-09 20:54:15,342 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-top 2023-02-09 20:54:15,347 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/.tmp/cf2/2cf0c2b26b2e41238e820322bdb43f97 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/2cf0c2b26b2e41238e820322bdb43f97 2023-02-09 20:54:15,348 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/.tmp/cf1/64b747c2f0434fa1a6628d55fe5b6c9b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/64b747c2f0434fa1a6628d55fe5b6c9b 2023-02-09 20:54:15,351 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5 2023-02-09 20:54:15,358 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881 2023-02-09 20:54:15,363 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e 2023-02-09 20:54:15,363 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(310): Store=11c5f86a4b18d67a5f0febb048d48b9f/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:15,364 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,365 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2 2023-02-09 20:54:15,366 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2 2023-02-09 20:54:15,366 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11c5f86a4b18d67a5f0febb048d48b9f columnFamilyName cf2 2023-02-09 20:54:15,368 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 81b4f32bb42c8db9e70dbb1fb15261b9/cf2 of 81b4f32bb42c8db9e70dbb1fb15261b9 into 2cf0c2b26b2e41238e820322bdb43f97(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:15,368 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 81b4f32bb42c8db9e70dbb1fb15261b9/cf1 of 81b4f32bb42c8db9e70dbb1fb15261b9 into 64b747c2f0434fa1a6628d55fe5b6c9b(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:15,370 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 81b4f32bb42c8db9e70dbb1fb15261b9: 2023-02-09 20:54:15,370 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 81b4f32bb42c8db9e70dbb1fb15261b9: 2023-02-09 20:54:15,370 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9., storeName=81b4f32bb42c8db9e70dbb1fb15261b9/cf2, priority=12, startTime=1675976055277; duration=0sec 2023-02-09 20:54:15,370 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9., storeName=81b4f32bb42c8db9e70dbb1fb15261b9/cf1, priority=12, startTime=1675976055276; duration=0sec 2023-02-09 20:54:15,371 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,371 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,382 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-top 2023-02-09 20:54:15,387 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-top 2023-02-09 20:54:15,392 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-top 2023-02-09 20:54:15,398 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-top 2023-02-09 20:54:15,402 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9 2023-02-09 20:54:15,407 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977 2023-02-09 20:54:15,413 DEBUG [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c 2023-02-09 20:54:15,413 INFO [StoreOpener-11c5f86a4b18d67a5f0febb048d48b9f-1] regionserver.HStore(310): Store=11c5f86a4b18d67a5f0febb048d48b9f/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:15,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:15,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:15,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 11c5f86a4b18d67a5f0febb048d48b9f; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72376805, jitterRate=0.07849843800067902}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:15,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 11c5f86a4b18d67a5f0febb048d48b9f: Running coprocessor pre-open hook at 1675976055283Writing region info on filesystem at 1675976055283Initializing all the Stores at 1675976055285 (+2 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976055286 (+1 ms)Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976055286Cleaning up temporary data from old regions at 1675976055417 (+131 ms)Cleaning up detritus from prior splits at 1675976055417Running coprocessor post-open hooks at 1675976055421 (+4 ms)Region opened successfully at 1675976055421 2023-02-09 20:54:15,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f., pid=17, masterSystemTime=1675976055159 2023-02-09 20:54:15,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,424 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:15,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:15,426 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,426 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:15,426 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1912): 11c5f86a4b18d67a5f0febb048d48b9f/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:15,426 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,426 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 11c5f86a4b18d67a5f0febb048d48b9f/cf1 in testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,426 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1912): 11c5f86a4b18d67a5f0febb048d48b9f/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:15,426 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 11c5f86a4b18d67a5f0febb048d48b9f/cf2 in testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,427 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/.tmp, totalSize=34.9 K 2023-02-09 20:54:15,427 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/.tmp, totalSize=34.9 K 2023-02-09 20:54:15,428 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976052273 2023-02-09 20:54:15,428 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976052273 2023-02-09 20:54:15,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:15,429 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=11c5f86a4b18d67a5f0febb048d48b9f, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,429 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976052633 2023-02-09 20:54:15,430 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055429"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976055429"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976055429"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976055429"}]},"ts":"1675976055429"} 2023-02-09 20:54:15,430 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976052633 2023-02-09 20:54:15,430 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976052885 2023-02-09 20:54:15,431 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976052885 2023-02-09 20:54:15,431 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976053121 2023-02-09 20:54:15,432 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976053359 2023-02-09 20:54:15,432 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976053121 2023-02-09 20:54:15,433 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976053359 2023-02-09 20:54:15,433 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976053594 2023-02-09 20:54:15,433 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976053594 2023-02-09 20:54:15,434 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976053821 2023-02-09 20:54:15,435 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976053821 2023-02-09 20:54:15,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-02-09 20:54:15,438 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure 11c5f86a4b18d67a5f0febb048d48b9f, server=jenkins-hbase9.apache.org,36111,1675976048792 in 426 msec 2023-02-09 20:54:15,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=12 2023-02-09 20:54:15,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, ASSIGN in 589 msec 2023-02-09 20:54:15,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f in 1.5980 sec 2023-02-09 20:54:15,464 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): 11c5f86a4b18d67a5f0febb048d48b9f#cf2#compaction#16 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:15,479 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): 11c5f86a4b18d67a5f0febb048d48b9f#cf1#compaction#17 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:15,487 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/.tmp/cf2/b895bb2fd8934b669c03514dcf39eb55 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/b895bb2fd8934b669c03514dcf39eb55 2023-02-09 20:54:15,493 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 11c5f86a4b18d67a5f0febb048d48b9f/cf2 of 11c5f86a4b18d67a5f0febb048d48b9f into b895bb2fd8934b669c03514dcf39eb55(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:15,494 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 11c5f86a4b18d67a5f0febb048d48b9f: 2023-02-09 20:54:15,494 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f., storeName=11c5f86a4b18d67a5f0febb048d48b9f/cf2, priority=9, startTime=1675976055425; duration=0sec 2023-02-09 20:54:15,494 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,499 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/.tmp/cf1/ad17ace443bb4a65a2fadfec3f6388e7 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/ad17ace443bb4a65a2fadfec3f6388e7 2023-02-09 20:54:15,507 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 11c5f86a4b18d67a5f0febb048d48b9f/cf1 of 11c5f86a4b18d67a5f0febb048d48b9f into ad17ace443bb4a65a2fadfec3f6388e7(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:15,507 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 11c5f86a4b18d67a5f0febb048d48b9f: 2023-02-09 20:54:15,507 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f., storeName=11c5f86a4b18d67a5f0febb048d48b9f/cf1, priority=9, startTime=1675976055423; duration=0sec 2023-02-09 20:54:15,507 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:15,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36111] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Scan size: 157 connection: 172.31.2.10:50954 deadline: 1675976115603, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099. is not online on jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,720 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-02-09 20:54:15,730 INFO [Listener at localhost/45551] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-09 20:54:15,733 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-02-09 20:54:15,762 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:15,772 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionUnevenDaughter 2023-02-09 20:54:15,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitTableRegionUnevenDaughter 2023-02-09 20:54:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:15,787 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976055787"}]},"ts":"1675976055787"} 2023-02-09 20:54:15,789 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLING in hbase:meta 2023-02-09 20:54:15,791 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionUnevenDaughter to state=DISABLING 2023-02-09 20:54:15,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, UNASSIGN}, {pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, UNASSIGN}] 2023-02-09 20:54:15,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-09 20:54:15,794 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, UNASSIGN 2023-02-09 20:54:15,795 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=11c5f86a4b18d67a5f0febb048d48b9f, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,795 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055795"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976055795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976055795"}]},"ts":"1675976055795"} 2023-02-09 20:54:15,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 11c5f86a4b18d67a5f0febb048d48b9f, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:15,803 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, UNASSIGN 2023-02-09 20:54:15,805 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=81b4f32bb42c8db9e70dbb1fb15261b9, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:15,805 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976055805"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976055805"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976055805"}]},"ts":"1675976055805"} 2023-02-09 20:54:15,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE; CloseRegionProcedure 81b4f32bb42c8db9e70dbb1fb15261b9, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:15,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-09 20:54:15,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:15,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 81b4f32bb42c8db9e70dbb1fb15261b9, disabling compactions & flushes 2023-02-09 20:54:15,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. after waiting 0 ms 2023-02-09 20:54:15,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:15,958 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-bottom] to archive 2023-02-09 20:54:15,959 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:15,966 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:15,968 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:15,970 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:15,972 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:15,999 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-bottom] to archive 2023-02-09 20:54:16,000 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:16,003 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,006 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,007 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,008 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,014 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:54:16,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:16,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9. 2023-02-09 20:54:16,015 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 81b4f32bb42c8db9e70dbb1fb15261b9: Waiting for close lock at 1675976055952Running coprocessor pre-close hooks at 1675976055952Disabling compacts and flushes for region at 1675976055952Disabling writes for close at 1675976055952Writing region close event to WAL at 1675976056009 (+57 ms)Running coprocessor post-close hooks at 1675976056015 (+6 ms)Closed at 1675976056015 2023-02-09 20:54:16,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:16,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:16,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 11c5f86a4b18d67a5f0febb048d48b9f, disabling compactions & flushes 2023-02-09 20:54:16,017 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:16,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:16,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. after waiting 0 ms 2023-02-09 20:54:16,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:16,018 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=81b4f32bb42c8db9e70dbb1fb15261b9, regionState=CLOSED 2023-02-09 20:54:16,018 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976056018"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976056018"}]},"ts":"1675976056018"} 2023-02-09 20:54:16,019 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5] to archive 2023-02-09 20:54:16,020 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:16,023 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/c16d4e70eed845099ea7e34073532c8d.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=20 2023-02-09 20:54:16,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; CloseRegionProcedure 81b4f32bb42c8db9e70dbb1fb15261b9, server=jenkins-hbase9.apache.org,36111,1675976048792 in 213 msec 2023-02-09 20:54:16,024 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7010ec35ff7a4a9186371b31ea0f4cf5.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,025 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=81b4f32bb42c8db9e70dbb1fb15261b9, UNASSIGN in 231 msec 2023-02-09 20:54:16,026 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/7753f0813ed847adaee39a2e8cf2afcc.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,028 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/0d94c953d302489690d45a22a3b65560.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,029 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-fb081348553d4ed899daff38f1a5ff6e 2023-02-09 20:54:16,031 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-cf2985c841e14522a8fc37296f348881 2023-02-09 20:54:16,032 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-570ad7f97d7447108a891e74a3532db5 2023-02-09 20:54:16,035 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977] to archive 2023-02-09 20:54:16,036 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:16,038 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,040 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/e49c7332f8d747db94e3aff1ebb1176e.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,041 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/1d7dc89f423c4e8b813ae100f5dc6839.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,043 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/72a95a7e95df448fbcf0577870d9eba0.ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,044 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-0a72afd3e738431baa01ed8764e5eba9 2023-02-09 20:54:16,045 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-bd1b6f0f339a47aaa3f2e48595f07f8c 2023-02-09 20:54:16,047 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/testSplitTableRegionUnevenDaughter=ec5090c9a8a7e77ccbbcbcb4ed761099-97bae105c5f04274974ed6e31c13a977 2023-02-09 20:54:16,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:54:16,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:16,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f. 2023-02-09 20:54:16,054 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 11c5f86a4b18d67a5f0febb048d48b9f: Waiting for close lock at 1675976056017Running coprocessor pre-close hooks at 1675976056017Disabling compacts and flushes for region at 1675976056017Disabling writes for close at 1675976056017Writing region close event to WAL at 1675976056048 (+31 ms)Running coprocessor post-close hooks at 1675976056054 (+6 ms)Closed at 1675976056054 2023-02-09 20:54:16,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:16,057 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=11c5f86a4b18d67a5f0febb048d48b9f, regionState=CLOSED 2023-02-09 20:54:16,057 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1675976056057"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976056057"}]},"ts":"1675976056057"} 2023-02-09 20:54:16,061 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-02-09 20:54:16,061 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 11c5f86a4b18d67a5f0febb048d48b9f, server=jenkins-hbase9.apache.org,36111,1675976048792 in 262 msec 2023-02-09 20:54:16,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=19 2023-02-09 20:54:16,064 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=11c5f86a4b18d67a5f0febb048d48b9f, UNASSIGN in 269 msec 2023-02-09 20:54:16,065 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976056065"}]},"ts":"1675976056065"} 2023-02-09 20:54:16,066 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLED in hbase:meta 2023-02-09 20:54:16,068 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionUnevenDaughter to state=DISABLED 2023-02-09 20:54:16,070 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter in 292 msec 2023-02-09 20:54:16,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-02-09 20:54:16,101 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 19 completed 2023-02-09 20:54:16,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,113 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,115 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=24, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-09 20:54:16,122 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,123 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:16,123 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:16,127 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/recovered.edits] 2023-02-09 20:54:16,127 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/recovered.edits] 2023-02-09 20:54:16,127 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/recovered.edits] 2023-02-09 20:54:16,131 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/64b747c2f0434fa1a6628d55fe5b6c9b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf1/64b747c2f0434fa1a6628d55fe5b6c9b 2023-02-09 20:54:16,132 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/ad17ace443bb4a65a2fadfec3f6388e7 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf1/ad17ace443bb4a65a2fadfec3f6388e7 2023-02-09 20:54:16,133 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/0d94c953d302489690d45a22a3b65560 2023-02-09 20:54:16,135 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/570ad7f97d7447108a891e74a3532db5 2023-02-09 20:54:16,136 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/2cf0c2b26b2e41238e820322bdb43f97 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/cf2/2cf0c2b26b2e41238e820322bdb43f97 2023-02-09 20:54:16,136 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/b895bb2fd8934b669c03514dcf39eb55 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/cf2/b895bb2fd8934b669c03514dcf39eb55 2023-02-09 20:54:16,137 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7010ec35ff7a4a9186371b31ea0f4cf5 2023-02-09 20:54:16,139 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/7753f0813ed847adaee39a2e8cf2afcc 2023-02-09 20:54:16,141 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9/recovered.edits/90.seqid 2023-02-09 20:54:16,141 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/c16d4e70eed845099ea7e34073532c8d 2023-02-09 20:54:16,141 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f/recovered.edits/90.seqid 2023-02-09 20:54:16,143 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/81b4f32bb42c8db9e70dbb1fb15261b9 2023-02-09 20:54:16,143 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:16,144 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/cf2985c841e14522a8fc37296f348881 2023-02-09 20:54:16,146 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf1/fb081348553d4ed899daff38f1a5ff6e 2023-02-09 20:54:16,149 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/0a72afd3e738431baa01ed8764e5eba9 2023-02-09 20:54:16,150 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:54:16,151 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/1d7dc89f423c4e8b813ae100f5dc6839 2023-02-09 20:54:16,152 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/72a95a7e95df448fbcf0577870d9eba0 2023-02-09 20:54:16,154 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/97bae105c5f04274974ed6e31c13a977 2023-02-09 20:54:16,156 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/bd1b6f0f339a47aaa3f2e48595f07f8c 2023-02-09 20:54:16,157 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/e49c7332f8d747db94e3aff1ebb1176e 2023-02-09 20:54:16,159 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/cf2/f2d82a72dba34a69b15cb4de7a7cfb4f 2023-02-09 20:54:16,162 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099/recovered.edits/85.seqid 2023-02-09 20:54:16,163 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionUnevenDaughter/ec5090c9a8a7e77ccbbcbcb4ed761099 2023-02-09 20:54:16,167 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-02-09 20:54:16,171 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=24, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,204 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionUnevenDaughter from hbase:meta 2023-02-09 20:54:16,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-09 20:54:16,219 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-09 20:54:16,220 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-09 20:54:16,249 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionUnevenDaughter' descriptor. 2023-02-09 20:54:16,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=24, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionUnevenDaughter' from region states. 2023-02-09 20:54:16,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976056251"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:16,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976056251"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:16,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976056251"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:16,254 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:54:16,254 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ec5090c9a8a7e77ccbbcbcb4ed761099, NAME => 'testSplitTableRegionUnevenDaughter,,1675976051762.ec5090c9a8a7e77ccbbcbcb4ed761099.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 81b4f32bb42c8db9e70dbb1fb15261b9, NAME => 'testSplitTableRegionUnevenDaughter,,1675976053845.81b4f32bb42c8db9e70dbb1fb15261b9.', STARTKEY => '', ENDKEY => '26'}, {ENCODED => 11c5f86a4b18d67a5f0febb048d48b9f, NAME => 'testSplitTableRegionUnevenDaughter,26,1675976053845.11c5f86a4b18d67a5f0febb048d48b9f.', STARTKEY => '26', ENDKEY => ''}] 2023-02-09 20:54:16,254 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionUnevenDaughter' as deleted. 2023-02-09 20:54:16,255 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976056255"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:16,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionUnevenDaughter state from META 2023-02-09 20:54:16,259 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=24, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:16,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter in 153 msec 2023-02-09 20:54:16,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-02-09 20:54:16,419 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 24 completed 2023-02-09 20:54:16,439 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=392 (was 358) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56722 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56726 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56682 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:48424 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:41648 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:1;jenkins-hbase9:36111-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:48174 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:39124 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:42058 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56700 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:48074 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:48052 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:48138 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56740 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56698 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:56710 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:42066 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=833 (was 693) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=256 (was 278), ProcessCount=168 (was 168), AvailableMemoryMB=10945 (was 11036) 2023-02-09 20:54:16,454 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=392, OpenFileDescriptor=833, MaxFileDescriptor=60000, SystemLoadAverage=256, ProcessCount=168, AvailableMemoryMB=10944 2023-02-09 20:54:16,454 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:16,464 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:16,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:16,467 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:16,469 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:16,470 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:16,472 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:16,473 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e empty. 2023-02-09 20:54:16,473 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:16,473 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-02-09 20:54:16,886 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:16,887 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(7671): creating {ENCODED => 65a23bd5b36d215caa9daae544c2807e, NAME => 'testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitWithoutPONR', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:16,899 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:16,899 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1603): Closing 65a23bd5b36d215caa9daae544c2807e, disabling compactions & flushes 2023-02-09 20:54:16,899 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:16,899 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:16,900 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. after waiting 0 ms 2023-02-09 20:54:16,900 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:16,900 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:16,900 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1557): Region close journal for 65a23bd5b36d215caa9daae544c2807e: Waiting for close lock at 1675976056899Disabling compacts and flushes for region at 1675976056899Disabling writes for close at 1675976056900 (+1 ms)Writing region close event to WAL at 1675976056900Closed at 1675976056900 2023-02-09 20:54:16,903 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:16,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976056904"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976056904"}]},"ts":"1675976056904"} 2023-02-09 20:54:16,906 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:16,907 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:16,908 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976056908"}]},"ts":"1675976056908"} 2023-02-09 20:54:16,909 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLING in hbase:meta 2023-02-09 20:54:16,913 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:16,915 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:16,915 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:16,915 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:16,915 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:16,915 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, ASSIGN}] 2023-02-09 20:54:16,917 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, ASSIGN 2023-02-09 20:54:16,918 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,39321,1675976048743; forceNewPlan=false, retain=false 2023-02-09 20:54:17,068 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:17,070 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,070 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976057070"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976057070"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976057070"}]},"ts":"1675976057070"} 2023-02-09 20:54:17,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:17,225 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,225 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 20:54:17,228 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34912, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 20:54:17,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 65a23bd5b36d215caa9daae544c2807e, NAME => 'testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:17,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:17,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:17,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,234 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,236 DEBUG [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1 2023-02-09 20:54:17,236 DEBUG [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1 2023-02-09 20:54:17,236 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65a23bd5b36d215caa9daae544c2807e columnFamilyName cf1 2023-02-09 20:54:17,237 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] regionserver.HStore(310): Store=65a23bd5b36d215caa9daae544c2807e/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:17,237 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,239 DEBUG [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2 2023-02-09 20:54:17,239 DEBUG [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2 2023-02-09 20:54:17,240 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 65a23bd5b36d215caa9daae544c2807e columnFamilyName cf2 2023-02-09 20:54:17,240 INFO [StoreOpener-65a23bd5b36d215caa9daae544c2807e-1] regionserver.HStore(310): Store=65a23bd5b36d215caa9daae544c2807e/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:17,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:17,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:17,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:17,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 65a23bd5b36d215caa9daae544c2807e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69148934, jitterRate=0.030399411916732788}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:17,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 65a23bd5b36d215caa9daae544c2807e: Running coprocessor pre-open hook at 1675976057233Writing region info on filesystem at 1675976057233Initializing all the Stores at 1675976057234 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976057234Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976057234Cleaning up temporary data from old regions at 1675976057242 (+8 ms)Cleaning up detritus from prior splits at 1675976057243 (+1 ms)Running coprocessor post-open hooks at 1675976057250 (+7 ms)Region opened successfully at 1675976057250 2023-02-09 20:54:17,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e., pid=27, masterSystemTime=1675976057225 2023-02-09 20:54:17,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,257 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,257 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976057256"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976057256"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976057256"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976057256"}]},"ts":"1675976057256"} 2023-02-09 20:54:17,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-02-09 20:54:17,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 in 187 msec 2023-02-09 20:54:17,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-02-09 20:54:17,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, ASSIGN in 346 msec 2023-02-09 20:54:17,264 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:17,264 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976057264"}]},"ts":"1675976057264"} 2023-02-09 20:54:17,265 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLED in hbase:meta 2023-02-09 20:54:17,268 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:17,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR in 802 msec 2023-02-09 20:54:17,470 DEBUG [Listener at localhost/45551] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-09 20:54:17,475 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:34914, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-09 20:54:17,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:17,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:17,484 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:17,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:17,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:17,486 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,486 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,488 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,488 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,488 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,488 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,489 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,489 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,490 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,490 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,490 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,490 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,491 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:17,492 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,492 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,493 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,493 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:54:17,493 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,493 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,494 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,504 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/ce31e78c7a654181b5fbab0d71f71848 2023-02-09 20:54:17,519 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/7df235e474c847d2a8217552cd10bfb1 2023-02-09 20:54:17,524 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/ce31e78c7a654181b5fbab0d71f71848 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 2023-02-09 20:54:17,529 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:17,530 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/7df235e474c847d2a8217552cd10bfb1 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 2023-02-09 20:54:17,535 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:17,536 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 43ms, sequenceid=6, compaction requested=false 2023-02-09 20:54:17,536 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitWithoutPONR' 2023-02-09 20:54:17,537 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:17,537 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,537 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:17,537 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:17,537 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,537 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,537 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,539 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:17,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30aa3975[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:17,541 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:17,541 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,544 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,544 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,544 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:17,546 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,546 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,546 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,547 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,547 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,547 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,547 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:17,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:17,552 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:17,552 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:17,552 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:17,552 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:17,552 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:17,553 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,553 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,553 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,553 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,553 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,554 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,555 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,555 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,556 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,557 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,557 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,560 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:17,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,561 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:17,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:17,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,662 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:17,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:17,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:17,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:17,682 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:17,682 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:17,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:17,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,687 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,687 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,687 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,688 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:17,689 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:17,689 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,690 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:17,690 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:17,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,691 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,691 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,691 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,700 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/e482ebf203af47e0a3d5607630eb7e0e 2023-02-09 20:54:17,714 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/fd33d2ef77a643789f97618cea3d5f9d 2023-02-09 20:54:17,719 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/e482ebf203af47e0a3d5607630eb7e0e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e 2023-02-09 20:54:17,720 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-09 20:54:17,721 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-09 20:54:17,721 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:17,721 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro Metrics about HBase RegionObservers 2023-02-09 20:54:17,724 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:17,724 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/fd33d2ef77a643789f97618cea3d5f9d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d 2023-02-09 20:54:17,729 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:17,730 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 40ms, sequenceid=19, compaction requested=false 2023-02-09 20:54:17,730 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:17,730 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,730 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:17,730 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:17,730 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,730 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,730 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,733 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,733 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,733 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,733 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:17,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28764429[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:17,735 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:17,735 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,737 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:17,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,737 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,737 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:17,738 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,738 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,738 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,745 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,745 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,745 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,746 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,746 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,746 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,747 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:17,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:17,753 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:17,753 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:17,754 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:17,754 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:17,754 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:17,755 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,755 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,755 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,756 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,757 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,757 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,760 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,760 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,761 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,764 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,765 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,765 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,765 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,765 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,765 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:17,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:17,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,777 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,777 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,777 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,777 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:17,777 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:17,877 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:17,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:17,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:17,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:17,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:17,902 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:17,902 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:17,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:17,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,906 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,907 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,907 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:17,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:17,908 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,908 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,909 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,909 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,909 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:17,909 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:17,910 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,910 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,910 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:17,911 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:17,912 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,913 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,913 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,913 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,913 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,913 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,925 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 2023-02-09 20:54:17,939 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/a90d64bfb94e416cb79e7a4af35782ca 2023-02-09 20:54:17,944 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 2023-02-09 20:54:17,950 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:17,951 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/a90d64bfb94e416cb79e7a4af35782ca as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca 2023-02-09 20:54:17,956 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:17,957 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 46ms, sequenceid=32, compaction requested=true 2023-02-09 20:54:17,957 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:17,957 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:17,957 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:17,957 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:17,957 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,957 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:17,957 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:17,959 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,959 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,959 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,959 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217bfa9[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:17,961 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:17,961 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:17,964 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:17,966 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,967 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,966 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,967 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,968 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:17,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:17,972 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:17,972 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:17,972 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:17,972 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:17,972 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,975 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:17,974 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,975 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,976 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:17,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,978 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,978 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:17,978 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,981 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,981 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,981 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,981 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,982 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:17,982 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,981 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,982 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,982 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:17,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:17,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:17,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:17,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,982 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:17,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:17,982 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:17,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:17,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:17,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:17,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,084 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:18,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:18,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:18,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:18,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:18,107 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:18,107 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,107 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:18,107 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:18,108 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,108 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,110 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,111 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,111 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:18,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,114 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,114 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,115 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,115 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,115 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,115 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,116 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,116 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,116 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,116 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:18,118 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:18,118 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:18,118 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,118 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,118 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,116 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,118 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:18,118 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,122 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,123 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,123 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,123 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,123 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,123 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,132 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/dddbd7adcf05464dab005a9350c4d3d0 2023-02-09 20:54:18,145 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/fe9b176d58524c5fb2e3af4549822b80 2023-02-09 20:54:18,151 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/dddbd7adcf05464dab005a9350c4d3d0 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 2023-02-09 20:54:18,157 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:18,157 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/fe9b176d58524c5fb2e3af4549822b80 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 2023-02-09 20:54:18,163 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:18,164 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 46ms, sequenceid=45, compaction requested=true 2023-02-09 20:54:18,164 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:18,164 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,164 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:18,164 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:18,164 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,164 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,164 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,166 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,166 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,166 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,166 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325944b4[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:18,168 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:18,168 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,170 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,170 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:18,170 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:18,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,171 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:18,172 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,172 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,172 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,174 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,174 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,174 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,175 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,175 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,175 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,175 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:18,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:18,180 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:18,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:18,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:18,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:18,180 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,183 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,183 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,183 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,185 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,186 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,186 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:18,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,205 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:18,206 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,205 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,207 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,307 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:18,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:18,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:18,329 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:18,329 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,329 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:18,329 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:18,330 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,330 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,331 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,331 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,331 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,331 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,332 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,332 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,334 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,334 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,334 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,334 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,334 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,335 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,336 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,336 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,337 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:18,337 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:18,337 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,337 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,337 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:18,337 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:18,338 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,338 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,355 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da 2023-02-09 20:54:18,374 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/2b0591e86c5f43db922a23a22384e569 2023-02-09 20:54:18,379 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da 2023-02-09 20:54:18,384 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:18,384 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/2b0591e86c5f43db922a23a22384e569 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 2023-02-09 20:54:18,389 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:18,390 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 53ms, sequenceid=58, compaction requested=true 2023-02-09 20:54:18,390 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:18,390 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,390 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:18,390 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:18,390 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,390 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,391 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,392 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,392 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,393 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:18,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53d9b21b[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:18,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:18,395 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:18,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:18,396 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:18,396 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,397 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,397 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,400 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,400 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,400 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,401 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,401 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,401 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,401 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:18,404 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:18,404 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:18,404 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:18,404 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:18,404 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,406 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,407 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,407 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,407 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,408 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,410 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,410 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,410 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,416 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,416 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:18,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,417 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,418 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,418 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,517 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:18,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-02-09 20:54:18,539 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-02-09 20:54:18,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-02-09 20:54:18,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-02-09 20:54:18,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,542 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,542 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-02-09 20:54:18,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,543 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-02-09 20:54:18,544 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,544 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,544 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:18,544 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:18,544 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,544 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. started... 2023-02-09 20:54:18,544 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:18,545 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,547 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,547 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,561 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/631f0688a80542f0ab3e1195d583b518 2023-02-09 20:54:18,576 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/6b15d74830f04a62919fbca2325fd0e8 2023-02-09 20:54:18,581 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/631f0688a80542f0ab3e1195d583b518 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 2023-02-09 20:54:18,586 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:18,587 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/6b15d74830f04a62919fbca2325fd0e8 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 2023-02-09 20:54:18,592 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:18,593 INFO [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 49ms, sequenceid=71, compaction requested=true 2023-02-09 20:54:18,593 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 65a23bd5b36d215caa9daae544c2807e: 2023-02-09 20:54:18,593 DEBUG [rs(jenkins-hbase9.apache.org,39321,1675976048743)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:18,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:18,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:18,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,594 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-02-09 20:54:18,594 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-02-09 20:54:18,595 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,595 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,596 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,596 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-02-09 20:54:18,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4994f345[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:18,597 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-02-09 20:54:18,598 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,599 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:18,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:18,599 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:18,601 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,601 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,603 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-02-09 20:54:18,603 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,603 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-02-09 20:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase9.apache.org,39321,1675976048743': 2023-02-09 20:54:18,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-02-09 20:54:18,606 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-02-09 20:54:18,606 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:18,606 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:18,606 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-02-09 20:54:18,606 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,607 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,608 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:18,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,608 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,608 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,609 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,610 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,610 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,611 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-02-09 20:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-02-09 20:54:18,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,613 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,613 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,614 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,615 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,615 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:18,615 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:18,615 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-02-09 20:54:18,615 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-02-09 20:54:18,714 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:18,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:18,980 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-09 20:54:18,981 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:18,983 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:18,984 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:18,984 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:18,984 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:18,985 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:18,985 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:18,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:18,985 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:19,233 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,234 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:19,234 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:19,235 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:19,236 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-02-09 20:54:19,237 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:19,237 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:19,237 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:19,237 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:19,238 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:19,241 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-09 20:54:19,243 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:19,243 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:19,244 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:19,244 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:19,244 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,245 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:19,245 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,245 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,246 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:19,246 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,246 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:19,246 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,246 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,246 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,247 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-02-09 20:54:19,247 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:19,248 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:19,249 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976089249 2023-02-09 20:54:19,250 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,250 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:19,255 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:19,256 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:19,256 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:19,256 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976119256 2023-02-09 20:54:19,256 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976179256 2023-02-09 20:54:19,256 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-09 20:54:19,257 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:19,259 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:19,259 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:19,259 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:19,507 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,507 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:19,507 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:19,507 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:19,507 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:19,508 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:19,508 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:19,508 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:19,508 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:19,509 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:19,511 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-09 20:54:19,511 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:19,512 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:19,512 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:19,513 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:19,513 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,514 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:19,514 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,514 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,515 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:19,515 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,515 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:19,515 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,516 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,516 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,516 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-02-09 20:54:19,517 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:19,518 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:19,519 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976089519 2023-02-09 20:54:19,520 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:19,520 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,523 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:19,524 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:19,524 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:19,524 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976119524 2023-02-09 20:54:19,524 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976179524 2023-02-09 20:54:19,524 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-09 20:54:19,525 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:19,530 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN}] 2023-02-09 20:54:19,530 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:19,530 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,530 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:19,531 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:19,775 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,775 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:19,775 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:19,775 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:19,776 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:19,776 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:19,776 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:19,776 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:19,776 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:19,777 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:19,779 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-09 20:54:19,780 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:19,780 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:19,781 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:19,781 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:19,782 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,782 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:19,782 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,783 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:19,783 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:19,783 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,784 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:19,784 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,784 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,784 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,784 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:54:19,785 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:19,785 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:19,785 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976089785 2023-02-09 20:54:19,787 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:19,787 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:19,789 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:19,789 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:19,790 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:19,790 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976119790 2023-02-09 20:54:19,790 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976179790 2023-02-09 20:54:19,790 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:19,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN}] 2023-02-09 20:54:19,790 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:19,792 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:19,793 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:19,793 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976059793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976059793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976059793"}]},"ts":"1675976059793"} 2023-02-09 20:54:19,794 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:19,795 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:19,795 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:19,795 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:19,795 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:20,040 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,040 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:20,040 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:20,040 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:20,041 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:20,041 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:20,041 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:20,041 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:20,041 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:20,042 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:20,043 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:20,043 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:20,044 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:20,044 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:20,044 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:20,045 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:20,045 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,045 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:20,046 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,046 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,046 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:20,047 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,047 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,047 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,047 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,047 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,048 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,048 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,048 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,048 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:20,048 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:20,049 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:20,049 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e to restore RIT 2023-02-09 20:54:20,049 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976090049 2023-02-09 20:54:20,050 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:20,050 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,053 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:20,053 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSING, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:20,053 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:20,053 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976120053 2023-02-09 20:54:20,053 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976180053 2023-02-09 20:54:20,053 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:20,054 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,054 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976060054"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976060054"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976060054"}]},"ts":"1675976060054"} 2023-02-09 20:54:20,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:20,056 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:20,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 65a23bd5b36d215caa9daae544c2807e, disabling compactions & flushes 2023-02-09 20:54:20,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:20,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:20,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. after waiting 0 ms 2023-02-09 20:54:20,210 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:20,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 65a23bd5b36d215caa9daae544c2807e 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:54:20,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/4d24fb09cb604e9585c852321075ae10 2023-02-09 20:54:20,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/f335a55389014089a67340940c59912f 2023-02-09 20:54:20,246 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf1/4d24fb09cb604e9585c852321075ae10 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 2023-02-09 20:54:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:20,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.tmp/cf2/f335a55389014089a67340940c59912f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f 2023-02-09 20:54:20,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:20,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 50ms, sequenceid=82, compaction requested=true 2023-02-09 20:54:20,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:54:20,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:20,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. 2023-02-09 20:54:20,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 65a23bd5b36d215caa9daae544c2807e: Waiting for close lock at 1675976060209Running coprocessor pre-close hooks at 1675976060209Disabling compacts and flushes for region at 1675976060209Disabling writes for close at 1675976060210 (+1 ms)Obtaining lock to block concurrent updates at 1675976060210Preparing flush snapshotting stores in 65a23bd5b36d215caa9daae544c2807e at 1675976060210Finished memstore snapshotting testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976060210Flushing stores of testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. at 1675976060210Flushing 65a23bd5b36d215caa9daae544c2807e/cf1: creating writer at 1675976060211 (+1 ms)Flushing 65a23bd5b36d215caa9daae544c2807e/cf1: appending metadata at 1675976060213 (+2 ms)Flushing 65a23bd5b36d215caa9daae544c2807e/cf1: closing flushed file at 1675976060213Flushing 65a23bd5b36d215caa9daae544c2807e/cf2: creating writer at 1675976060227 (+14 ms)Flushing 65a23bd5b36d215caa9daae544c2807e/cf2: appending metadata at 1675976060229 (+2 ms)Flushing 65a23bd5b36d215caa9daae544c2807e/cf2: closing flushed file at 1675976060229Flushing 65a23bd5b36d215caa9daae544c2807e/cf1: reopening flushed file at 1675976060247 (+18 ms)Flushing 65a23bd5b36d215caa9daae544c2807e/cf2: reopening flushed file at 1675976060254 (+7 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 65a23bd5b36d215caa9daae544c2807e in 50ms, sequenceid=82, compaction requested=true at 1675976060260 (+6 ms)Writing region close event to WAL at 1675976060277 (+17 ms)Running coprocessor post-close hooks at 1675976060282 (+5 ms)Closed at 1675976060282 2023-02-09 20:54:20,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,285 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED 2023-02-09 20:54:20,285 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976060285"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976060285"}]},"ts":"1675976060285"} 2023-02-09 20:54:20,286 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:20,286 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:20,287 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:20,304 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:20,304 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:20,305 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:20,306 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:20,306 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-09 20:54:20,306 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-02-09 20:54:20,306 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:20,307 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:20,307 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:20,307 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:20,308 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,308 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:20,308 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,308 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,309 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:20,309 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,309 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,309 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,310 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,310 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,310 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,310 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,310 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,310 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,310 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,310 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:20,311 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:20,311 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:20,311 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e to restore RIT 2023-02-09 20:54:20,311 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976090311 2023-02-09 20:54:20,312 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:20,312 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,314 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:20,314 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:20,314 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:20,314 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976120314 2023-02-09 20:54:20,315 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976180315 2023-02-09 20:54:20,315 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-09 20:54:20,315 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,315 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976060315"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976060315"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976060315"}]},"ts":"1675976060315"} 2023-02-09 20:54:20,317 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:20,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-02-09 20:54:20,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 in 261 msec 2023-02-09 20:54:20,319 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:20,319 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:20,319 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:20,565 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,565 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:20,565 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:20,565 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:20,566 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:20,566 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:20,566 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:20,566 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:20,566 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:20,567 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:20,568 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:20,568 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-09 20:54:20,568 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:20,569 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:20,569 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:20,570 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:20,570 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:20,570 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,570 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:20,571 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,571 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,571 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:20,572 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,572 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,572 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:20,572 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:20,572 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:20,573 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:20,573 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:20,574 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e to restore RIT 2023-02-09 20:54:20,574 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976090574 2023-02-09 20:54:20,574 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:20,575 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,577 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:20,577 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:20,577 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:20,577 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976120577 2023-02-09 20:54:20,578 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976180578 2023-02-09 20:54:20,578 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:20,578 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,580 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:20,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-02-09 20:54:20,583 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN in 790 msec 2023-02-09 20:54:20,586 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:20,586 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,586 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:20,586 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:20,828 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:20,828 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:20,829 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:20,829 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:20,829 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:20,830 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:20,831 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:20,831 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:20,831 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:20,831 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:20,832 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:20,832 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:20,832 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,832 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:20,833 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,833 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:20,833 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:20,834 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:20,834 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:20,835 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:20,835 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:20,835 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976090835 2023-02-09 20:54:20,836 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:20,837 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,839 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:20,839 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:20,839 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:20,839 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976120839 2023-02-09 20:54:20,839 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976180839 2023-02-09 20:54:20,840 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:20,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,840 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:20,845 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:20,854 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=65a23bd5b36d215caa9daae544c2807e, threads=14 2023-02-09 20:54:20,855 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,855 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,855 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,855 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,856 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,857 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,860 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,860 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,861 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,973 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,974 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,977 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,977 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,980 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,980 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:20,999 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,003 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,004 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,004 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,008 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,009 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,009 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,347 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,347 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 65a23bd5b36d215caa9daae544c2807e Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:21,397 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:21,397 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:21,397 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:21,518 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:54:21,590 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,590 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:21,590 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:21,590 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:21,590 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:21,591 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:21,591 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:21,591 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:21,591 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:21,592 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:21,593 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-09 20:54:21,593 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:21,593 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:21,594 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:21,594 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:21,594 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:21,595 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:21,595 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:21,596 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:21,596 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:21,597 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:21,597 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:21,598 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,598 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:21,599 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:21,599 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:21,599 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,599 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,599 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,599 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:21,600 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:21,600 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-02-09 20:54:21,600 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:21,601 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:21,601 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976091601 2023-02-09 20:54:21,603 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,603 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:21,606 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:21,607 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:21,607 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:21,607 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976121607 2023-02-09 20:54:21,607 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976181607 2023-02-09 20:54:21,607 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:21,621 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=65a23bd5b36d215caa9daae544c2807e, threads=14 2023-02-09 20:54:21,622 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,622 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,622 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,622 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,623 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,627 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,627 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,627 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,627 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,628 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,628 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,628 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,628 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,628 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,724 WARN [DataStreamer for file /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/.splits/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-09 20:54:21,822 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,833 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,846 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,876 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,883 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,883 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,885 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,885 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,885 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,907 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,915 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,915 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,915 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,918 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 for region: 65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:21,918 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 65a23bd5b36d215caa9daae544c2807e Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:21,958 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:21,964 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:21,965 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:21,966 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:21,966 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:21,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:21,966 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:22,108 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,108 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:22,108 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:22,108 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:22,108 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:22,109 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:22,109 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:22,109 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:22,109 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:22,109 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:22,110 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-09 20:54:22,111 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:22,111 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:22,111 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:22,111 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:22,111 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:22,112 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:22,112 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,112 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:22,112 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,113 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,113 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:22,114 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,114 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,114 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:22,114 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:22,114 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:22,115 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:22,115 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:22,116 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976092116 2023-02-09 20:54:22,117 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,117 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:22,120 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:22,120 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:22,120 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:22,120 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976122120 2023-02-09 20:54:22,120 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976182120 2023-02-09 20:54:22,120 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:22,123 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:22,125 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:22,125 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:22,126 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:22,371 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:22,371 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:22,371 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:22,371 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:22,372 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:22,372 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:22,372 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:22,373 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:22,373 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:22,373 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:22,374 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-09 20:54:22,374 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:22,374 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:22,375 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:22,375 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:22,375 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:22,376 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:22,376 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,376 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:22,376 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,377 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:22,377 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:22,377 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,377 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN 2023-02-09 20:54:22,378 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,378 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:22,378 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,378 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,378 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,378 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=65a23bd5b36d215caa9daae544c2807e, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:22,378 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 65a23bd5b36d215caa9daae544c2807e, server=jenkins-hbase9.apache.org,39321,1675976048743 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:22,378 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:22,379 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:22,379 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:22,380 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976092380 2023-02-09 20:54:22,381 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:22,381 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,383 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:22,383 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=65a23bd5b36d215caa9daae544c2807e, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,39321,1675976048743, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743, openSeqNum=2 2023-02-09 20:54:22,383 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:22,383 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976122383 2023-02-09 20:54:22,383 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976182383 2023-02-09 20:54:22,383 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-09 20:54:22,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976062385"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1675976062385"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1675976062385"}]},"ts":"1675976062385"} 2023-02-09 20:54:22,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062385"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976062385"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976062385"}]},"ts":"1675976062385"} 2023-02-09 20:54:22,386 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062385"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976062385"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976062385"}]},"ts":"1675976062385"} 2023-02-09 20:54:22,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, ASSIGN}, {pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, ASSIGN}] 2023-02-09 20:54:22,392 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, ASSIGN 2023-02-09 20:54:22,393 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,39321,1675976048743; forceNewPlan=false, retain=false 2023-02-09 20:54:22,394 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, ASSIGN 2023-02-09 20:54:22,395 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,39321,1675976048743; forceNewPlan=false, retain=false 2023-02-09 20:54:22,543 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:54:22,545 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=7f4003ce5139ea977370b6c09c1a1a3b, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,545 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976062545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976062545"}]},"ts":"1675976062545"} 2023-02-09 20:54:22,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 7f4003ce5139ea977370b6c09c1a1a3b, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:22,550 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=53cdc9bd760aa9b9473ca264cd521151, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,551 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062550"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976062550"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976062550"}]},"ts":"1675976062550"} 2023-02-09 20:54:22,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=32, state=RUNNABLE; OpenRegionProcedure 53cdc9bd760aa9b9473ca264cd521151, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:22,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 53cdc9bd760aa9b9473ca264cd521151, NAME => 'testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.', STARTKEY => '41', ENDKEY => ''} 2023-02-09 20:54:22,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:22,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:22,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,706 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,707 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1 2023-02-09 20:54:22,707 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1 2023-02-09 20:54:22,707 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53cdc9bd760aa9b9473ca264cd521151 columnFamilyName cf1 2023-02-09 20:54:22,724 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-top 2023-02-09 20:54:22,728 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-top 2023-02-09 20:54:22,733 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-top 2023-02-09 20:54:22,739 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-top 2023-02-09 20:54:22,744 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-top 2023-02-09 20:54:22,749 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-top 2023-02-09 20:54:22,754 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-top 2023-02-09 20:54:22,755 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(310): Store=53cdc9bd760aa9b9473ca264cd521151/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:22,755 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,756 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2 2023-02-09 20:54:22,756 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2 2023-02-09 20:54:22,756 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53cdc9bd760aa9b9473ca264cd521151 columnFamilyName cf2 2023-02-09 20:54:22,774 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-top 2023-02-09 20:54:22,779 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-top 2023-02-09 20:54:22,785 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-top 2023-02-09 20:54:22,791 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-top 2023-02-09 20:54:22,797 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-top 2023-02-09 20:54:22,802 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-top 2023-02-09 20:54:22,809 DEBUG [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-top 2023-02-09 20:54:22,810 INFO [StoreOpener-53cdc9bd760aa9b9473ca264cd521151-1] regionserver.HStore(310): Store=53cdc9bd760aa9b9473ca264cd521151/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:22,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:22,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:22,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 53cdc9bd760aa9b9473ca264cd521151; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69641162, jitterRate=0.03773418068885803}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:22,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 53cdc9bd760aa9b9473ca264cd521151: Running coprocessor pre-open hook at 1675976062704Writing region info on filesystem at 1675976062704Initializing all the Stores at 1675976062705 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976062705Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976062705Cleaning up temporary data from old regions at 1675976062813 (+108 ms)Cleaning up detritus from prior splits at 1675976062814 (+1 ms)Running coprocessor post-open hooks at 1675976062818 (+4 ms)Region opened successfully at 1675976062818 2023-02-09 20:54:22,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151., pid=34, masterSystemTime=1675976062699 2023-02-09 20:54:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:22,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:22,821 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:22,821 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:22,821 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,821 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1912): 53cdc9bd760aa9b9473ca264cd521151/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:22,821 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 53cdc9bd760aa9b9473ca264cd521151/cf2 in testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,822 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/.tmp, totalSize=34.9 K 2023-02-09 20:54:22,821 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,822 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1912): 53cdc9bd760aa9b9473ca264cd521151/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:22,823 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 53cdc9bd760aa9b9473ca264cd521151/cf1 in testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,823 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/.tmp, totalSize=34.9 K 2023-02-09 20:54:22,823 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976057475 2023-02-09 20:54:22,824 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976057475 2023-02-09 20:54:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:22,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,824 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976057664 2023-02-09 20:54:22,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7f4003ce5139ea977370b6c09c1a1a3b, NAME => 'testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.', STARTKEY => '', ENDKEY => '41'} 2023-02-09 20:54:22,825 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976057664 2023-02-09 20:54:22,825 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976057880 2023-02-09 20:54:22,825 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=53cdc9bd760aa9b9473ca264cd521151, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:22,825 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062825"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976062825"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976062825"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976062825"}]},"ts":"1675976062825"} 2023-02-09 20:54:22,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:22,826 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976058087 2023-02-09 20:54:22,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,826 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976058310 2023-02-09 20:54:22,827 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976057880 2023-02-09 20:54:22,827 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976058520 2023-02-09 20:54:22,828 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976058087 2023-02-09 20:54:22,828 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976058717 2023-02-09 20:54:22,829 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976058310 2023-02-09 20:54:22,831 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976058520 2023-02-09 20:54:22,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=32 2023-02-09 20:54:22,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=32, state=SUCCESS; OpenRegionProcedure 53cdc9bd760aa9b9473ca264cd521151, server=jenkins-hbase9.apache.org,39321,1675976048743 in 276 msec 2023-02-09 20:54:22,833 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976058717 2023-02-09 20:54:22,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, ASSIGN in 442 msec 2023-02-09 20:54:22,843 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,845 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1 2023-02-09 20:54:22,845 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1 2023-02-09 20:54:22,846 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f4003ce5139ea977370b6c09c1a1a3b columnFamilyName cf1 2023-02-09 20:54:22,846 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] throttle.PressureAwareThroughputController(145): 53cdc9bd760aa9b9473ca264cd521151#cf2#compaction#32 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:22,891 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53cdc9bd760aa9b9473ca264cd521151#cf1#compaction#33 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:22,895 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/.tmp/cf2/338d8de7f91b40799a0948c9ac64ae2d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/338d8de7f91b40799a0948c9ac64ae2d 2023-02-09 20:54:22,897 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-bottom 2023-02-09 20:54:22,903 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-bottom 2023-02-09 20:54:22,903 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 53cdc9bd760aa9b9473ca264cd521151/cf2 of 53cdc9bd760aa9b9473ca264cd521151 into 338d8de7f91b40799a0948c9ac64ae2d(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:22,903 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 53cdc9bd760aa9b9473ca264cd521151: 2023-02-09 20:54:22,903 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151., storeName=53cdc9bd760aa9b9473ca264cd521151/cf2, priority=9, startTime=1675976062820; duration=0sec 2023-02-09 20:54:22,903 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:22,914 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-bottom 2023-02-09 20:54:22,920 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-bottom 2023-02-09 20:54:22,930 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/.tmp/cf1/e287e7e3f4e349f693946536da69d0eb as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e287e7e3f4e349f693946536da69d0eb 2023-02-09 20:54:22,932 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-bottom 2023-02-09 20:54:22,936 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-bottom 2023-02-09 20:54:22,937 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 53cdc9bd760aa9b9473ca264cd521151/cf1 of 53cdc9bd760aa9b9473ca264cd521151 into e287e7e3f4e349f693946536da69d0eb(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:22,937 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 53cdc9bd760aa9b9473ca264cd521151: 2023-02-09 20:54:22,937 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151., storeName=53cdc9bd760aa9b9473ca264cd521151/cf1, priority=9, startTime=1675976062819; duration=0sec 2023-02-09 20:54:22,937 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:22,941 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-bottom 2023-02-09 20:54:22,941 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(310): Store=7f4003ce5139ea977370b6c09c1a1a3b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:22,941 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,942 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2 2023-02-09 20:54:22,942 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2 2023-02-09 20:54:22,942 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7f4003ce5139ea977370b6c09c1a1a3b columnFamilyName cf2 2023-02-09 20:54:22,960 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-bottom 2023-02-09 20:54:22,964 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-bottom 2023-02-09 20:54:22,968 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-bottom 2023-02-09 20:54:22,973 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-bottom 2023-02-09 20:54:22,980 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-bottom 2023-02-09 20:54:22,984 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-bottom 2023-02-09 20:54:22,988 DEBUG [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-bottom 2023-02-09 20:54:22,988 INFO [StoreOpener-7f4003ce5139ea977370b6c09c1a1a3b-1] regionserver.HStore(310): Store=7f4003ce5139ea977370b6c09c1a1a3b/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:22,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:22,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:22,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 7f4003ce5139ea977370b6c09c1a1a3b; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61616053, jitterRate=-0.08184926211833954}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:22,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 7f4003ce5139ea977370b6c09c1a1a3b: Running coprocessor pre-open hook at 1675976062826Writing region info on filesystem at 1675976062826Initializing all the Stores at 1675976062827 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976062827Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976062843 (+16 ms)Cleaning up temporary data from old regions at 1675976062990 (+147 ms)Cleaning up detritus from prior splits at 1675976062990Running coprocessor post-open hooks at 1675976062993 (+3 ms)Region opened successfully at 1675976062993 2023-02-09 20:54:22,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b., pid=33, masterSystemTime=1675976062699 2023-02-09 20:54:22,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:22,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:22,995 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:22,995 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:22,995 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,995 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,995 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1912): 7f4003ce5139ea977370b6c09c1a1a3b/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:22,995 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1912): 7f4003ce5139ea977370b6c09c1a1a3b/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:22,995 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 7f4003ce5139ea977370b6c09c1a1a3b/cf1 in testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,995 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 7f4003ce5139ea977370b6c09c1a1a3b/cf2 in testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,996 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/.tmp, totalSize=34.9 K 2023-02-09 20:54:22,996 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/.tmp, totalSize=34.9 K 2023-02-09 20:54:22,996 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976057475 2023-02-09 20:54:22,996 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976057475 2023-02-09 20:54:22,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:22,997 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976057664 2023-02-09 20:54:22,997 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976057664 2023-02-09 20:54:22,997 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976057880 2023-02-09 20:54:22,997 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976057880 2023-02-09 20:54:22,998 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976058087 2023-02-09 20:54:22,998 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976058087 2023-02-09 20:54:22,998 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976058310 2023-02-09 20:54:22,998 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976058310 2023-02-09 20:54:22,998 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=7f4003ce5139ea977370b6c09c1a1a3b, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:22,999 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976062998"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976062998"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976062998"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976062998"}]},"ts":"1675976062998"} 2023-02-09 20:54:22,999 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting 6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976058520 2023-02-09 20:54:22,999 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976058520 2023-02-09 20:54:22,999 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] compactions.Compactor(207): Compacting f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976058717 2023-02-09 20:54:22,999 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] compactions.Compactor(207): Compacting 4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976058717 2023-02-09 20:54:23,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-02-09 20:54:23,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 7f4003ce5139ea977370b6c09c1a1a3b, server=jenkins-hbase9.apache.org,39321,1675976048743 in 454 msec 2023-02-09 20:54:23,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=28 2023-02-09 20:54:23,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, ASSIGN in 615 msec 2023-02-09 20:54:23,009 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 in 4.0250 sec 2023-02-09 20:54:23,016 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] throttle.PressureAwareThroughputController(145): 7f4003ce5139ea977370b6c09c1a1a3b#cf2#compaction#34 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:23,016 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7f4003ce5139ea977370b6c09c1a1a3b#cf1#compaction#35 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:23,031 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/.tmp/cf1/2d744aea2a1c4a24b601fe829b677bf6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/2d744aea2a1c4a24b601fe829b677bf6 2023-02-09 20:54:23,035 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/.tmp/cf2/62e9b78b15e44e37b7362eada55c7464 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/62e9b78b15e44e37b7362eada55c7464 2023-02-09 20:54:23,037 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7f4003ce5139ea977370b6c09c1a1a3b/cf1 of 7f4003ce5139ea977370b6c09c1a1a3b into 2d744aea2a1c4a24b601fe829b677bf6(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:23,037 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 7f4003ce5139ea977370b6c09c1a1a3b: 2023-02-09 20:54:23,038 INFO [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b., storeName=7f4003ce5139ea977370b6c09c1a1a3b/cf1, priority=9, startTime=1675976062994; duration=0sec 2023-02-09 20:54:23,038 DEBUG [RS:0;jenkins-hbase9:39321-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:23,042 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7f4003ce5139ea977370b6c09c1a1a3b/cf2 of 7f4003ce5139ea977370b6c09c1a1a3b into 62e9b78b15e44e37b7362eada55c7464(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:23,042 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 7f4003ce5139ea977370b6c09c1a1a3b: 2023-02-09 20:54:23,042 INFO [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b., storeName=7f4003ce5139ea977370b6c09c1a1a3b/cf2, priority=9, startTime=1675976062995; duration=0sec 2023-02-09 20:54:23,042 DEBUG [RS:0;jenkins-hbase9:39321-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:23,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39321] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Scan size: 143 connection: 172.31.2.10:34914 deadline: 1675976123135, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e. is not online on jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:23,249 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:54:23,257 INFO [Listener at localhost/45551] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-09 20:54:23,260 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:54:23,277 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:23,278 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitWithoutPONR 2023-02-09 20:54:23,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitWithoutPONR 2023-02-09 20:54:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-09 20:54:23,283 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976063283"}]},"ts":"1675976063283"} 2023-02-09 20:54:23,285 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLING in hbase:meta 2023-02-09 20:54:23,286 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitWithoutPONR to state=DISABLING 2023-02-09 20:54:23,288 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, UNASSIGN}, {pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, UNASSIGN}] 2023-02-09 20:54:23,288 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, UNASSIGN 2023-02-09 20:54:23,289 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=53cdc9bd760aa9b9473ca264cd521151, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:23,289 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976063289"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976063289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976063289"}]},"ts":"1675976063289"} 2023-02-09 20:54:23,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; CloseRegionProcedure 53cdc9bd760aa9b9473ca264cd521151, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:23,293 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, UNASSIGN 2023-02-09 20:54:23,293 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7f4003ce5139ea977370b6c09c1a1a3b, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:23,294 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976063293"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976063293"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976063293"}]},"ts":"1675976063293"} 2023-02-09 20:54:23,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE; CloseRegionProcedure 7f4003ce5139ea977370b6c09c1a1a3b, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-09 20:54:23,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:23,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 7f4003ce5139ea977370b6c09c1a1a3b, disabling compactions & flushes 2023-02-09 20:54:23,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:23,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:23,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. after waiting 0 ms 2023-02-09 20:54:23,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:23,456 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-bottom] to archive 2023-02-09 20:54:23,457 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:23,458 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,460 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,461 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,462 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,463 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,464 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,465 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,467 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-bottom] to archive 2023-02-09 20:54:23,468 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:23,469 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,470 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,471 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,473 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,474 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,475 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,476 DEBUG [StoreCloser-testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:54:23,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:23,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b. 2023-02-09 20:54:23,481 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 7f4003ce5139ea977370b6c09c1a1a3b: Waiting for close lock at 1675976063443Running coprocessor pre-close hooks at 1675976063443Disabling compacts and flushes for region at 1675976063443Disabling writes for close at 1675976063443Writing region close event to WAL at 1675976063477 (+34 ms)Running coprocessor post-close hooks at 1675976063480 (+3 ms)Closed at 1675976063480 2023-02-09 20:54:23,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:23,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:23,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 53cdc9bd760aa9b9473ca264cd521151, disabling compactions & flushes 2023-02-09 20:54:23,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:23,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:23,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. after waiting 0 ms 2023-02-09 20:54:23,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:23,483 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7f4003ce5139ea977370b6c09c1a1a3b, regionState=CLOSED 2023-02-09 20:54:23,483 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976063483"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976063483"}]},"ts":"1675976063483"} 2023-02-09 20:54:23,484 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10-top] to archive 2023-02-09 20:54:23,484 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:23,486 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=36 2023-02-09 20:54:23,487 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; CloseRegionProcedure 7f4003ce5139ea977370b6c09c1a1a3b, server=jenkins-hbase9.apache.org,39321,1675976048743 in 189 msec 2023-02-09 20:54:23,488 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=7f4003ce5139ea977370b6c09c1a1a3b, UNASSIGN in 199 msec 2023-02-09 20:54:23,489 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,490 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,492 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,493 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,494 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,495 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f-top] to archive 2023-02-09 20:54:23,496 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:23,497 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,498 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,499 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,500 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,501 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,502 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,503 DEBUG [StoreCloser-testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:54:23,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:23,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151. 2023-02-09 20:54:23,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 53cdc9bd760aa9b9473ca264cd521151: Waiting for close lock at 1675976063482Running coprocessor pre-close hooks at 1675976063482Disabling compacts and flushes for region at 1675976063482Disabling writes for close at 1675976063482Writing region close event to WAL at 1675976063504 (+22 ms)Running coprocessor post-close hooks at 1675976063507 (+3 ms)Closed at 1675976063507 2023-02-09 20:54:23,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:23,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=53cdc9bd760aa9b9473ca264cd521151, regionState=CLOSED 2023-02-09 20:54:23,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976063509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976063509"}]},"ts":"1675976063509"} 2023-02-09 20:54:23,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=37 2023-02-09 20:54:23,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=37, state=SUCCESS; CloseRegionProcedure 53cdc9bd760aa9b9473ca264cd521151, server=jenkins-hbase9.apache.org,39321,1675976048743 in 220 msec 2023-02-09 20:54:23,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=35 2023-02-09 20:54:23,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=53cdc9bd760aa9b9473ca264cd521151, UNASSIGN in 224 msec 2023-02-09 20:54:23,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976063515"}]},"ts":"1675976063515"} 2023-02-09 20:54:23,516 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLED in hbase:meta 2023-02-09 20:54:23,518 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitWithoutPONR to state=DISABLED 2023-02-09 20:54:23,519 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR in 239 msec 2023-02-09 20:54:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-02-09 20:54:23,585 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitWithoutPONR, procId: 35 completed 2023-02-09 20:54:23,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitWithoutPONR 2023-02-09 20:54:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,589 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-02-09 20:54:23,589 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=40, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,595 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,595 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:23,595 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:23,598 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/recovered.edits] 2023-02-09 20:54:23,598 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/recovered.edits] 2023-02-09 20:54:23,598 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/recovered.edits] 2023-02-09 20:54:23,602 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/.regioninfo to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/.regioninfo 2023-02-09 20:54:23,602 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/.regioninfo to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/.regioninfo 2023-02-09 20:54:23,602 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/4d24fb09cb604e9585c852321075ae10 2023-02-09 20:54:23,604 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6 2023-02-09 20:54:23,605 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,605 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/631f0688a80542f0ab3e1195d583b518 2023-02-09 20:54:23,605 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/4d24fb09cb604e9585c852321075ae10.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,606 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,606 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da 2023-02-09 20:54:23,607 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/5ce903ca5cb64357aa620ff0a0dfd6d6.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,608 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,608 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/631f0688a80542f0ab3e1195d583b518.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,608 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/ce31e78c7a654181b5fbab0d71f71848 2023-02-09 20:54:23,609 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,609 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/a4835c9fbeb34ea9b64a0f9dcf2227da.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,610 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/dddbd7adcf05464dab005a9350c4d3d0 2023-02-09 20:54:23,610 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,611 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/ce31e78c7a654181b5fbab0d71f71848.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,611 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf1/e482ebf203af47e0a3d5607630eb7e0e 2023-02-09 20:54:23,612 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,612 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/dddbd7adcf05464dab005a9350c4d3d0.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,613 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,614 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf1/e482ebf203af47e0a3d5607630eb7e0e.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,614 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/2b0591e86c5f43db922a23a22384e569 2023-02-09 20:54:23,616 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/6b15d74830f04a62919fbca2325fd0e8 2023-02-09 20:54:23,617 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,618 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/7df235e474c847d2a8217552cd10bfb1 2023-02-09 20:54:23,618 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/2b0591e86c5f43db922a23a22384e569.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,619 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/a90d64bfb94e416cb79e7a4af35782ca 2023-02-09 20:54:23,619 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,620 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/6b15d74830f04a62919fbca2325fd0e8.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,621 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/f335a55389014089a67340940c59912f 2023-02-09 20:54:23,622 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,622 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/7df235e474c847d2a8217552cd10bfb1.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,623 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fd33d2ef77a643789f97618cea3d5f9d 2023-02-09 20:54:23,623 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,623 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/a90d64bfb94e416cb79e7a4af35782ca.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,625 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/cf2/fe9b176d58524c5fb2e3af4549822b80 2023-02-09 20:54:23,625 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,626 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/f335a55389014089a67340940c59912f.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,628 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,628 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/fd33d2ef77a643789f97618cea3d5f9d.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,630 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/7f4003ce5139ea977370b6c09c1a1a3b/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,630 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e/recovered.edits/85.seqid 2023-02-09 20:54:23,630 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/53cdc9bd760aa9b9473ca264cd521151/cf2/fe9b176d58524c5fb2e3af4549822b80.65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,631 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/65a23bd5b36d215caa9daae544c2807e 2023-02-09 20:54:23,634 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/2d744aea2a1c4a24b601fe829b677bf6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf1/2d744aea2a1c4a24b601fe829b677bf6 2023-02-09 20:54:23,634 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e287e7e3f4e349f693946536da69d0eb to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf1/e287e7e3f4e349f693946536da69d0eb 2023-02-09 20:54:23,637 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/62e9b78b15e44e37b7362eada55c7464 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/cf2/62e9b78b15e44e37b7362eada55c7464 2023-02-09 20:54:23,638 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/338d8de7f91b40799a0948c9ac64ae2d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/cf2/338d8de7f91b40799a0948c9ac64ae2d 2023-02-09 20:54:23,641 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b/recovered.edits/90.seqid 2023-02-09 20:54:23,641 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151/recovered.edits/90.seqid 2023-02-09 20:54:23,642 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/7f4003ce5139ea977370b6c09c1a1a3b 2023-02-09 20:54:23,642 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitWithoutPONR/53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:23,642 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-02-09 20:54:23,645 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=40, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,648 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitWithoutPONR from hbase:meta 2023-02-09 20:54:23,651 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitWithoutPONR' descriptor. 2023-02-09 20:54:23,652 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=40, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,652 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitWithoutPONR' from region states. 2023-02-09 20:54:23,652 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976063652"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:23,652 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976063652"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:23,652 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976063652"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:23,654 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:54:23,654 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 65a23bd5b36d215caa9daae544c2807e, NAME => 'testSplitWithoutPONR,,1675976056466.65a23bd5b36d215caa9daae544c2807e.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 7f4003ce5139ea977370b6c09c1a1a3b, NAME => 'testSplitWithoutPONR,,1675976058982.7f4003ce5139ea977370b6c09c1a1a3b.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 53cdc9bd760aa9b9473ca264cd521151, NAME => 'testSplitWithoutPONR,41,1675976058982.53cdc9bd760aa9b9473ca264cd521151.', STARTKEY => '41', ENDKEY => ''}] 2023-02-09 20:54:23,654 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitWithoutPONR' as deleted. 2023-02-09 20:54:23,655 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976063655"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:23,656 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitWithoutPONR state from META 2023-02-09 20:54:23,659 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=40, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:23,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR in 73 msec 2023-02-09 20:54:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-02-09 20:54:23,691 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitWithoutPONR, procId: 40 completed 2023-02-09 20:54:23,707 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=403 (was 392) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:56722 [Waiting for operation #20] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e91e44f-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase9:39321-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:48424 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:48174 [Waiting for operation #37] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:34232 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:42058 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:56700 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:51926 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:48074 [Waiting for operation #33] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:52506 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:48052 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:42262 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:48138 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:42182 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:56740 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:56698 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1956816783_15 at /127.0.0.1:42066 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=865 (was 833) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=235 (was 256), ProcessCount=168 (was 168), AvailableMemoryMB=10822 (was 10944) 2023-02-09 20:54:23,725 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=403, OpenFileDescriptor=865, MaxFileDescriptor=60000, SystemLoadAverage=235, ProcessCount=168, AvailableMemoryMB=10822 2023-02-09 20:54:23,726 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:23,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:23,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:23,732 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:23,734 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:23,735 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:23,736 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:23,737 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 empty. 2023-02-09 20:54:23,738 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:23,738 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-02-09 20:54:23,753 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:23,755 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(7671): creating {ENCODED => d14b935178c82ede44119501cb287f81, NAME => 'testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:23,769 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:23,769 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1603): Closing d14b935178c82ede44119501cb287f81, disabling compactions & flushes 2023-02-09 20:54:23,769 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:23,769 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:23,769 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. after waiting 0 ms 2023-02-09 20:54:23,769 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:23,769 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:23,770 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1557): Region close journal for d14b935178c82ede44119501cb287f81: Waiting for close lock at 1675976063769Disabling compacts and flushes for region at 1675976063769Disabling writes for close at 1675976063769Writing region close event to WAL at 1675976063769Closed at 1675976063769 2023-02-09 20:54:23,773 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:23,774 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976063773"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976063773"}]},"ts":"1675976063773"} 2023-02-09 20:54:23,777 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:23,778 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:23,778 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976063778"}]},"ts":"1675976063778"} 2023-02-09 20:54:23,780 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2023-02-09 20:54:23,783 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:23,784 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:23,784 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:23,784 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:23,784 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:23,784 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN}] 2023-02-09 20:54:23,786 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:23,787 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:54:23,937 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:23,939 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:23,939 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976063939"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976063939"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976063939"}]},"ts":"1675976063939"} 2023-02-09 20:54:23,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:24,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d14b935178c82ede44119501cb287f81, NAME => 'testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:24,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:24,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:24,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,104 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,105 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1 2023-02-09 20:54:24,105 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1 2023-02-09 20:54:24,105 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d14b935178c82ede44119501cb287f81 columnFamilyName cf1 2023-02-09 20:54:24,106 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(310): Store=d14b935178c82ede44119501cb287f81/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:24,106 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,107 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2 2023-02-09 20:54:24,107 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2 2023-02-09 20:54:24,107 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d14b935178c82ede44119501cb287f81 columnFamilyName cf2 2023-02-09 20:54:24,108 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(310): Store=d14b935178c82ede44119501cb287f81/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:24,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:24,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:24,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:24,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened d14b935178c82ede44119501cb287f81; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69064448, jitterRate=0.029140472412109375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:24,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for d14b935178c82ede44119501cb287f81: Running coprocessor pre-open hook at 1675976064102Writing region info on filesystem at 1675976064102Initializing all the Stores at 1675976064103 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976064103Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976064103Cleaning up temporary data from old regions at 1675976064110 (+7 ms)Cleaning up detritus from prior splits at 1675976064110Running coprocessor post-open hooks at 1675976064114 (+4 ms)Region opened successfully at 1675976064114 2023-02-09 20:54:24,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81., pid=43, masterSystemTime=1675976064092 2023-02-09 20:54:24,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,117 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,117 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976064117"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976064117"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976064117"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976064117"}]},"ts":"1675976064117"} 2023-02-09 20:54:24,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-02-09 20:54:24,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 in 178 msec 2023-02-09 20:54:24,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=41 2023-02-09 20:54:24,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=41, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN in 337 msec 2023-02-09 20:54:24,124 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:24,124 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976064124"}]},"ts":"1675976064124"} 2023-02-09 20:54:24,126 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2023-02-09 20:54:24,131 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:24,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution in 401 msec 2023-02-09 20:54:24,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:24,243 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:24,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:24,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,244 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,245 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,245 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,246 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,246 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,247 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:24,248 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:24,248 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,248 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:24,249 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:54:24,249 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,250 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,250 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,250 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,250 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,250 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,261 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/f87245d60079446198f87b25997abe7f 2023-02-09 20:54:24,276 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/713fa478ca0246b7bc54a7cd22aae57a 2023-02-09 20:54:24,281 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/f87245d60079446198f87b25997abe7f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f 2023-02-09 20:54:24,285 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:24,286 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/713fa478ca0246b7bc54a7cd22aae57a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a 2023-02-09 20:54:24,290 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:24,291 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 42ms, sequenceid=6, compaction requested=false 2023-02-09 20:54:24,291 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,291 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:24,291 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,291 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:24,291 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:24,292 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,292 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,292 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,294 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,294 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,294 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,294 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:24,296 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7641f729[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:24,296 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,297 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,297 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,297 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,298 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,298 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:24,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,298 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,300 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,300 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:24,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:24,303 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:24,303 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:24,303 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:24,303 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:24,303 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:24,306 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,306 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,306 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,306 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,306 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,307 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,307 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,308 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,309 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,310 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,310 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,312 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,312 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,313 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,313 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:24,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,314 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,414 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:24,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:24,433 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,433 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,433 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:24,433 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:24,433 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,433 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,435 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,436 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,436 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:24,438 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,438 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:24,438 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,440 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,440 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,440 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,440 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,450 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/28be57de72d74a03b774b3d774cd61b5 2023-02-09 20:54:24,463 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/8e6231a0e2744cca865357497f3b485f 2023-02-09 20:54:24,467 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/28be57de72d74a03b774b3d774cd61b5 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 2023-02-09 20:54:24,471 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:24,472 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/8e6231a0e2744cca865357497f3b485f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f 2023-02-09 20:54:24,477 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:24,478 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 40ms, sequenceid=19, compaction requested=false 2023-02-09 20:54:24,478 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:24,478 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,478 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:24,478 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:24,478 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,478 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,478 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,481 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,481 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,481 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,481 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec571b1[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:24,483 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:24,483 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:24,486 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,490 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,492 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,492 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,492 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,492 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:24,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:24,495 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:24,495 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:24,495 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:24,495 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:24,495 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,498 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,498 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,500 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,500 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,501 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,503 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,503 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,504 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,505 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,604 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:24,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:24,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:24,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:24,621 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,621 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,622 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:24,622 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:24,622 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,622 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,623 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,623 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,623 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,623 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,624 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,625 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:24,626 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:24,626 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,626 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:24,626 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:24,628 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,629 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,629 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,636 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/b23cea0406f6447fa387b2a244101c66 2023-02-09 20:54:24,652 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/8108a083589f469bbaad0f3398c49244 2023-02-09 20:54:24,658 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/b23cea0406f6447fa387b2a244101c66 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 2023-02-09 20:54:24,663 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:24,664 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/8108a083589f469bbaad0f3398c49244 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 2023-02-09 20:54:24,669 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:24,670 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 44ms, sequenceid=32, compaction requested=true 2023-02-09 20:54:24,670 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:24,670 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,670 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:24,670 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:24,670 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,670 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,670 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,672 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,672 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,672 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,672 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10925318[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:24,674 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:24,674 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,676 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:24,676 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,679 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:24,683 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:24,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:24,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:24,683 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:24,683 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,685 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,686 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,687 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,688 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,690 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,690 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,690 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:24,696 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,696 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,697 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:24,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,697 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,697 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,698 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,797 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:24,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:24,815 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,815 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,815 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:24,815 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:24,816 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,816 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,818 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,818 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,818 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,818 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,819 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,819 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,820 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:24,821 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:24,821 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,821 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:24,821 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:24,822 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,831 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/46f09f90f4094bb5a32508da572a9423 2023-02-09 20:54:24,843 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/f3c7ee99e7e847a5aeb8d6317609942b 2023-02-09 20:54:24,847 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/46f09f90f4094bb5a32508da572a9423 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 2023-02-09 20:54:24,851 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:24,852 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/f3c7ee99e7e847a5aeb8d6317609942b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b 2023-02-09 20:54:24,856 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:24,857 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 36ms, sequenceid=45, compaction requested=true 2023-02-09 20:54:24,857 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:24,857 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:24,857 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:24,857 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:24,857 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,857 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:24,858 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:24,859 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,859 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,860 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,860 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef28494[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:24,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:24,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,864 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:24,864 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:24,864 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:24,866 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,866 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,866 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,866 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:24,869 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:24,870 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:24,870 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:24,870 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:24,870 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,872 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,872 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,873 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,874 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,874 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:24,875 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:24,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,879 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:24,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:24,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:24,880 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,980 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:24,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:24,996 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:24,996 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:24,996 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:24,996 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:24,997 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,997 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,998 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:24,998 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:24,999 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:24,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:25,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,000 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,000 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,000 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:25,001 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:25,002 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,003 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,003 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,003 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,003 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,003 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,010 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/7dc3695533a943f0bf00c5c86b493997 2023-02-09 20:54:25,022 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/46c1ebbdead04aa7897e535bf1e17df6 2023-02-09 20:54:25,028 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/7dc3695533a943f0bf00c5c86b493997 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 2023-02-09 20:54:25,032 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:25,033 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/46c1ebbdead04aa7897e535bf1e17df6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 2023-02-09 20:54:25,038 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:25,039 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 38ms, sequenceid=58, compaction requested=true 2023-02-09 20:54:25,039 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:25,039 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:25,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:25,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:25,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,042 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3ccdc[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:25,044 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:25,044 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,045 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:25,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,045 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:25,046 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,046 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,046 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:25,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,047 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,047 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,047 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,051 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:25,054 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:25,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:25,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:25,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:25,054 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:25,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,056 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,056 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,057 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,058 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,059 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,059 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,062 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,062 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:25,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,062 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,062 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:25,062 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,063 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,163 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:25,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:25,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-02-09 20:54:25,180 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-02-09 20:54:25,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-02-09 20:54:25,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:54:25,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,181 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,182 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,182 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,181 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,182 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,182 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-02-09 20:54:25,183 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,183 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,184 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:25,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-02-09 20:54:25,184 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. started... 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:25,184 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,186 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,187 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,187 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,187 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,187 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,187 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,195 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/abd8fc686ac642e8822f7e5bd3a4d796 2023-02-09 20:54:25,209 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/d091b7dbc3814593874eb7cce9f2c03a 2023-02-09 20:54:25,213 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/abd8fc686ac642e8822f7e5bd3a4d796 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 2023-02-09 20:54:25,217 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:25,218 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/d091b7dbc3814593874eb7cce9f2c03a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a 2023-02-09 20:54:25,221 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:25,222 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 38ms, sequenceid=71, compaction requested=true 2023-02-09 20:54:25,222 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:25,222 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:25,222 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:25,222 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:25,222 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,222 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-02-09 20:54:25,222 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-02-09 20:54:25,224 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,224 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,224 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,224 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-02-09 20:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229e6d6a[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:25,226 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:54:25,226 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,227 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:25,227 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:54:25,228 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-02-09 20:54:25,228 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:25,230 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,230 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,230 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,230 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,230 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,230 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,231 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-02-09 20:54:25,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-02-09 20:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:54:25,234 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-02-09 20:54:25,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:25,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:25,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-02-09 20:54:25,234 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,235 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,236 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:25,236 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:25,238 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-02-09 20:54:25,239 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,239 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:25,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-02-09 20:54:25,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,245 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:25,245 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-02-09 20:54:25,246 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-02-09 20:54:25,346 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:25,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:25,606 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-09 20:54:25,607 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:25,609 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,610 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,610 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,610 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,611 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:25,611 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:25,611 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:25,859 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,859 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:25,859 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:25,860 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:25,860 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-09 20:54:25,860 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:25,860 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:25,860 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:25,860 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:25,861 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:25,862 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-09 20:54:25,862 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:25,863 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:25,863 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:25,863 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:25,864 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:25,864 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:25,864 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:25,864 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:25,865 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:25,865 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:25,866 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:25,866 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:25,866 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:25,867 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,867 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:25,867 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,867 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,867 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,867 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:54:25,870 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:25,871 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:25,871 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976095871 2023-02-09 20:54:25,873 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:25,873 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,875 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:25,876 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:25,876 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:25,876 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976125876 2023-02-09 20:54:25,876 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976185876 2023-02-09 20:54:25,876 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:25,876 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:25,879 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:25,879 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:25,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:25,880 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:26,126 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:26,127 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:26,127 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:26,128 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:26,129 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-09 20:54:26,129 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:26,130 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:26,130 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:26,130 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:26,130 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,131 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:26,131 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,131 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,131 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,132 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:26,132 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,132 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,132 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:26,132 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,133 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,133 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,133 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,133 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,133 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:26,133 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:26,134 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:26,134 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976096134 2023-02-09 20:54:26,136 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,136 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:26,138 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,139 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,139 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:26,139 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976126139 2023-02-09 20:54:26,139 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976186139 2023-02-09 20:54:26,139 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:26,139 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:26,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN}] 2023-02-09 20:54:26,141 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:26,141 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:26,142 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:26,389 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,389 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:26,389 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:26,389 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:26,390 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:26,391 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:26,391 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:26,391 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:26,391 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:26,391 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:26,393 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-09 20:54:26,393 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:26,393 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:26,394 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:26,394 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:26,394 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,394 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:26,395 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,395 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,395 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,395 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:26,396 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,396 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,396 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:26,397 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,397 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,397 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,397 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,397 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,397 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:26,398 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:26,398 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:26,398 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976096398 2023-02-09 20:54:26,399 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:26,400 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,402 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,403 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,403 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:26,403 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976126403 2023-02-09 20:54:26,403 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976186403 2023-02-09 20:54:26,403 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:26,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN}] 2023-02-09 20:54:26,403 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:26,405 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,406 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,406 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976066406"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976066406"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976066406"}]},"ts":"1675976066406"} 2023-02-09 20:54:26,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:26,407 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:26,407 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:26,408 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:26,653 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,653 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:26,653 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:26,654 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:26,654 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:26,654 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:26,654 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:26,654 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:26,654 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:26,655 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:26,656 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:26,656 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:26,656 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:26,657 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:26,657 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:26,657 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:26,657 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,658 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:26,658 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,658 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,659 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,659 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:26,659 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,660 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,660 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:26,661 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,661 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,661 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,661 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,661 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,661 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,662 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,662 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,662 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:54:26,662 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:26,663 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:26,663 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:26,663 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976096663 2023-02-09 20:54:26,663 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:26,664 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,666 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,667 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSING, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,667 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:26,667 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976126667 2023-02-09 20:54:26,667 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976186667 2023-02-09 20:54:26,667 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:26,667 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,667 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976066667"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976066667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976066667"}]},"ts":"1675976066667"} 2023-02-09 20:54:26,669 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:26,669 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:26,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close d14b935178c82ede44119501cb287f81 2023-02-09 20:54:26,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing d14b935178c82ede44119501cb287f81, disabling compactions & flushes 2023-02-09 20:54:26,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:26,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:26,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. after waiting 0 ms 2023-02-09 20:54:26,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:26,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing d14b935178c82ede44119501cb287f81 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:54:26,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/12a72bff98c2492f9b1333dac479d273 2023-02-09 20:54:26,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/f4ca3de332b74955878a1ddee826ca31 2023-02-09 20:54:26,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/12a72bff98c2492f9b1333dac479d273 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 2023-02-09 20:54:26,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:26,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/f4ca3de332b74955878a1ddee826ca31 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 2023-02-09 20:54:26,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:54:26,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 52ms, sequenceid=82, compaction requested=true 2023-02-09 20:54:26,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:54:26,880 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:26,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:26,880 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for d14b935178c82ede44119501cb287f81: Waiting for close lock at 1675976066822Running coprocessor pre-close hooks at 1675976066822Disabling compacts and flushes for region at 1675976066822Disabling writes for close at 1675976066822Obtaining lock to block concurrent updates at 1675976066822Preparing flush snapshotting stores in d14b935178c82ede44119501cb287f81 at 1675976066822Finished memstore snapshotting testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976066822Flushing stores of testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. at 1675976066823 (+1 ms)Flushing d14b935178c82ede44119501cb287f81/cf1: creating writer at 1675976066824 (+1 ms)Flushing d14b935178c82ede44119501cb287f81/cf1: appending metadata at 1675976066826 (+2 ms)Flushing d14b935178c82ede44119501cb287f81/cf1: closing flushed file at 1675976066826Flushing d14b935178c82ede44119501cb287f81/cf2: creating writer at 1675976066844 (+18 ms)Flushing d14b935178c82ede44119501cb287f81/cf2: appending metadata at 1675976066846 (+2 ms)Flushing d14b935178c82ede44119501cb287f81/cf2: closing flushed file at 1675976066846Flushing d14b935178c82ede44119501cb287f81/cf1: reopening flushed file at 1675976066863 (+17 ms)Flushing d14b935178c82ede44119501cb287f81/cf2: reopening flushed file at 1675976066869 (+6 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for d14b935178c82ede44119501cb287f81 in 52ms, sequenceid=82, compaction requested=true at 1675976066874 (+5 ms)Writing region close event to WAL at 1675976066876 (+2 ms)Running coprocessor post-close hooks at 1675976066880 (+4 ms)Closed at 1675976066880 2023-02-09 20:54:26,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed d14b935178c82ede44119501cb287f81 2023-02-09 20:54:26,884 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSED 2023-02-09 20:54:26,884 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976066884"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976066884"}]},"ts":"1675976066884"} 2023-02-09 20:54:26,885 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:26,885 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,885 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:26,885 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:26,917 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,917 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:26,917 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:26,917 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:26,920 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:26,920 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:26,920 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:26,920 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:26,921 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:26,921 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:26,922 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:26,922 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-09 20:54:26,922 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-02-09 20:54:26,922 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:26,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:26,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:26,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:26,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:26,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:26,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,925 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:26,925 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,925 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:26,925 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:26,926 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,926 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,926 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,926 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,926 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,926 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,926 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:26,926 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,926 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,927 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:26,927 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:26,927 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:26,928 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:26,928 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:26,933 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976096933 2023-02-09 20:54:26,934 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,934 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:26,937 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,937 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:26,937 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:26,937 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976126937 2023-02-09 20:54:26,937 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976186937 2023-02-09 20:54:26,937 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:26,938 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSED, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:26,938 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976066938"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976066938"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976066938"}]},"ts":"1675976066938"} 2023-02-09 20:54:26,940 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:26,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=45 2023-02-09 20:54:26,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 in 271 msec 2023-02-09 20:54:26,943 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:26,943 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:26,944 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:26,944 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:27,188 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,188 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:27,188 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:27,188 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:27,189 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:27,189 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:27,189 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:27,189 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:27,189 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:27,190 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:27,191 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:27,191 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-02-09 20:54:27,191 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:27,192 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:27,192 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:27,192 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:27,192 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:27,193 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,193 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:27,193 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,193 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,194 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,194 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:27,194 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,194 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,194 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:27,195 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,195 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:27,195 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:27,195 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:27,196 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,196 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,196 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,196 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,196 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,196 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:27,196 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:27,197 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:27,197 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:27,197 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976097197 2023-02-09 20:54:27,197 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:27,198 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:27,200 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,201 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,201 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:27,201 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976127201 2023-02-09 20:54:27,201 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976187201 2023-02-09 20:54:27,201 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:27,201 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:27,202 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:27,205 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-02-09 20:54:27,205 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN in 799 msec 2023-02-09 20:54:27,208 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:27,208 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:27,208 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:27,451 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,451 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:27,451 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:27,451 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:27,452 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:27,452 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:27,452 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:27,452 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:27,453 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:27,453 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:27,454 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:27,454 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:27,454 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:27,454 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:27,455 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:27,455 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:27,455 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:27,455 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,456 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:27,456 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,456 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,457 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,457 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:27,457 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,457 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,458 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:27,458 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,459 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:27,459 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:27,459 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,459 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,459 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,459 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:27,460 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:27,460 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:27,461 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976097461 2023-02-09 20:54:27,461 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:27,462 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,464 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,464 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,465 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:27,465 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976127465 2023-02-09 20:54:27,465 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976187465 2023-02-09 20:54:27,465 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:27,465 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,465 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,465 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,470 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:27,478 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=d14b935178c82ede44119501cb287f81, threads=14 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,479 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,480 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,480 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,480 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,480 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,480 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,481 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,481 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,481 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,658 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,664 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,669 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,670 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,671 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,674 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,679 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,683 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,685 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,691 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,692 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,692 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,694 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,696 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,696 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region d14b935178c82ede44119501cb287f81 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:27,740 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:27,740 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:27,740 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:27,965 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,965 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:27,965 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:27,965 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:27,966 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:27,966 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:27,966 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:27,966 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:27,966 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:27,967 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:27,967 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:27,968 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:27,969 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,969 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:27,969 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,969 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:27,970 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,970 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:27,970 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,970 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:27,970 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:27,971 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,971 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,971 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,971 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:27,972 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:27,972 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:27,973 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:27,973 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976097973 2023-02-09 20:54:27,974 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:27,975 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:27,977 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,978 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:27,978 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:27,978 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976127978 2023-02-09 20:54:27,978 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976187978 2023-02-09 20:54:27,978 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:27,993 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=d14b935178c82ede44119501cb287f81, threads=14 2023-02-09 20:54:27,994 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,994 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,994 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,994 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,995 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,996 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,996 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,996 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:27,997 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,153 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,163 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,171 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,172 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,172 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,180 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,194 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,199 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,203 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,206 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:54:28,231 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,231 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,234 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,237 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,617 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f for region: d14b935178c82ede44119501cb287f81 2023-02-09 20:54:28,617 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region d14b935178c82ede44119501cb287f81 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:28,654 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:28,659 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/8eee9e859e7e19ef31af324e2d22b2c3/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:28,660 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/b2171a2584fed6219ade2983f2412e19/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:54:28,660 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:28,660 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:28,660 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:28,728 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,728 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:28,729 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:28,729 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:28,730 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:28,730 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:28,730 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:28,730 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:28,730 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:28,731 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:28,731 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-09 20:54:28,731 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:28,732 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:28,732 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:28,732 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:28,732 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:28,732 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:28,733 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,733 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:28,733 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,733 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,733 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,734 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:28,734 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,734 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,734 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:28,735 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,735 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:28,735 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:28,737 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:28,737 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,737 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,737 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,737 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:28,737 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:28,737 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:54:28,738 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:28,738 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:28,739 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976098739 2023-02-09 20:54:28,739 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:28,740 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,742 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:28,742 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:28,743 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:28,743 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976128743 2023-02-09 20:54:28,743 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976188743 2023-02-09 20:54:28,743 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:28,745 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:28,748 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:28,748 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:28,748 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:28,993 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 0 rollback state: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:28,994 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:28,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:28,995 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:28,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-09 20:54:28,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:28,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:28,996 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:28,996 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:28,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:28,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:28,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:28,999 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,999 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,999 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:28,999 DEBUG [Listener at localhost/45551] procedure2.StateMachineProcedure(225): Abort requested for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,000 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:29,000 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:29,001 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:29,001 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976099001 2023-02-09 20:54:29,001 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:29,004 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,006 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,006 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,006 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:29,006 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976129006 2023-02-09 20:54:29,006 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976189006 2023-02-09 20:54:29,006 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-09 20:54:29,007 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:29,019 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:29,019 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:29,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:29,019 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:29,257 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 1 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,257 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:29,258 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:29,258 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:29,258 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:29,258 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:29,258 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:29,258 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:29,259 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:29,259 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:29,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-02-09 20:54:29,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:29,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:29,261 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:29,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:29,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:29,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:29,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:29,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:29,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:29,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:29,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:29,266 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:29,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,267 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,267 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,267 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-02-09 20:54:29,267 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:29,268 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:29,268 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976099268 2023-02-09 20:54:29,270 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:29,271 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,273 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,274 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,274 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:29,274 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976129274 2023-02-09 20:54:29,274 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976189274 2023-02-09 20:54:29,274 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:29,274 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:29,276 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:29,276 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:29,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:29,276 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:29,525 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 2 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,525 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:29,525 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:29,525 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:29,526 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:29,526 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:29,526 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:29,526 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:29,526 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:29,527 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:29,528 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-09 20:54:29,528 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:29,528 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:29,528 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:29,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:29,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:29,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:29,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:29,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:29,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:29,532 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,532 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,532 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,532 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,532 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:29,533 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:29,533 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:29,534 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976099534 2023-02-09 20:54:29,534 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:29,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,537 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,537 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,537 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:29,537 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976129537 2023-02-09 20:54:29,537 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976189537 2023-02-09 20:54:29,538 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:29,538 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:29,540 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:29,540 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:29,540 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:29,540 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:29,788 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 3 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,788 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:29,788 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:29,788 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:29,789 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:29,789 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:29,789 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:29,789 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:29,789 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:29,790 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:29,791 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-09 20:54:29,791 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:29,791 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:29,791 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:29,792 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:29,792 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:29,792 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:29,792 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,792 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:29,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:29,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:29,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:29,794 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:29,794 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,795 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,795 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,795 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,795 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:29,795 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:29,795 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:29,796 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:29,796 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976099796 2023-02-09 20:54:29,797 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:29,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:29,800 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,800 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:29,800 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:29,800 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976129800 2023-02-09 20:54:29,800 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976189800 2023-02-09 20:54:29,800 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:29,800 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:29,804 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1054): Stored [pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN] 2023-02-09 20:54:29,804 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:29,804 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:29,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:29,804 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:30,051 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 4 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,051 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:30,051 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:30,051 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:30,052 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:30,052 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:30,052 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:30,052 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:30,052 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:30,053 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:30,054 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-09 20:54:30,054 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:54:30,054 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:54:30,054 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:30,055 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:30,055 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:30,055 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:30,055 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:30,056 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,056 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:30,056 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,056 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,057 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,057 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:30,057 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,057 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:30,059 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,059 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,059 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,059 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,060 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,060 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,060 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:54:30,060 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:30,061 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:30,061 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:30,063 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976100063 2023-02-09 20:54:30,063 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:30,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,066 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,068 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:30,068 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976130068 2023-02-09 20:54:30,068 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976190068 2023-02-09 20:54:30,068 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:30,069 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:30,073 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:30,073 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:30,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:30,073 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:30,319 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 5 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,319 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:30,319 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:30,319 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:30,320 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:30,320 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:30,320 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:30,320 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:30,320 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:30,321 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:30,322 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:30,323 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,323 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:30,323 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,323 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,323 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,324 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:30,324 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,324 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,324 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:30,324 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,325 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,325 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:30,325 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,325 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,325 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,325 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,325 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-02-09 20:54:30,326 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:30,326 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:30,326 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:30,326 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976100326 2023-02-09 20:54:30,327 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:30,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,329 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,331 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,331 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,331 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:30,331 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976130331 2023-02-09 20:54:30,331 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976190331 2023-02-09 20:54:30,331 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:30,331 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:30,333 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:30,333 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:30,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:30,333 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:30,581 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 6 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,581 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:30,581 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:30,582 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:30,582 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:30,582 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:30,582 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:30,582 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:30,582 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:30,583 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:30,583 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-09 20:54:30,584 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:30,584 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:30,584 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:30,584 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:30,584 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:30,585 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,585 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:30,585 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,585 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,585 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,586 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:30,587 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,587 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,587 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,587 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,587 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-02-09 20:54:30,587 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:30,588 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:30,588 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:30,588 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976100588 2023-02-09 20:54:30,590 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:30,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,591 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,593 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,594 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,594 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:30,594 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976130594 2023-02-09 20:54:30,594 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976190594 2023-02-09 20:54:30,594 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:54:30,594 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:30,597 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:30,597 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-02-09 20:54:30,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:30,597 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:30,845 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(509): Restart 7 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,845 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:30,845 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:30,845 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:30,847 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:30,847 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:30,847 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:30,847 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:30,847 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:30,848 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:30,849 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-09 20:54:30,849 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-02-09 20:54:30,849 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:30,849 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:30,850 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:30,850 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:30,850 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,850 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:30,851 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,851 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:30,851 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,852 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:30,852 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,852 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:30,852 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:30,853 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,853 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,853 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:30,853 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,853 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,853 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,853 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:54:30,853 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:30,854 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:30,854 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:30,855 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:30,855 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976100855 2023-02-09 20:54:30,855 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:30,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:30,859 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,861 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,861 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:30,861 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:30,861 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976130861 2023-02-09 20:54:30,861 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976190861 2023-02-09 20:54:30,861 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 5 msec 2023-02-09 20:54:30,861 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:54:30,863 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 exec-time=5.2540 sec 2023-02-09 20:54:30,863 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:30,864 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=true 2023-02-09 20:54:31,014 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:31,015 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:31,016 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976071015"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976071015"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976071015"}]},"ts":"1675976071015"} 2023-02-09 20:54:31,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:31,018 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:54:31,018 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:31,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:54:31,018 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:54:31,111 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:31,111 WARN [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(526): ProcedureExecutor not running, may have been stopped by pending procedure due to KillAndToggleBeforeStoreUpdate flag. 2023-02-09 20:54:31,111 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:54:31,111 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:54:31,111 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:54:31,112 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:54:31,112 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:54:31,112 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:54:31,112 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:54:31,112 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:54:31,113 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:54:31,114 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[0] 2023-02-09 20:54:31,114 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:54:31,115 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:54:31,115 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:54:31,115 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:54:31,115 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:31,116 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:54:31,116 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:31,116 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:54:31,117 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:31,117 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:54:31,117 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:31,117 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:54:31,118 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,118 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:54:31,118 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:31,118 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:54:31,118 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:31,119 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-02-09 20:54:31,119 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:54:31,120 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:54:31,120 INFO [Listener at localhost/45551] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81 to restore RIT 2023-02-09 20:54:31,120 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976101120 2023-02-09 20:54:31,120 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:54:31,121 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:54:31,123 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:31,123 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=d14b935178c82ede44119501cb287f81, regionState=OPENING, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:54:31,123 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:54:31,124 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976131124 2023-02-09 20:54:31,124 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976191124 2023-02-09 20:54:31,124 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:54:31,124 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:31,124 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976071124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976071124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976071124"}]},"ts":"1675976071124"} 2023-02-09 20:54:31,126 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:31,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d14b935178c82ede44119501cb287f81, NAME => 'testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:31,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:31,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:31,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,283 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,284 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1 2023-02-09 20:54:31,284 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1 2023-02-09 20:54:31,285 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d14b935178c82ede44119501cb287f81 columnFamilyName cf1 2023-02-09 20:54:31,291 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 2023-02-09 20:54:31,295 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 2023-02-09 20:54:31,298 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 2023-02-09 20:54:31,301 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 2023-02-09 20:54:31,305 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 2023-02-09 20:54:31,308 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 2023-02-09 20:54:31,314 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f 2023-02-09 20:54:31,314 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(310): Store=d14b935178c82ede44119501cb287f81/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:31,314 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,315 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2 2023-02-09 20:54:31,315 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2 2023-02-09 20:54:31,316 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d14b935178c82ede44119501cb287f81 columnFamilyName cf2 2023-02-09 20:54:31,322 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 2023-02-09 20:54:31,327 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a 2023-02-09 20:54:31,330 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 2023-02-09 20:54:31,333 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f 2023-02-09 20:54:31,336 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a 2023-02-09 20:54:31,340 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b 2023-02-09 20:54:31,343 DEBUG [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 2023-02-09 20:54:31,343 INFO [StoreOpener-d14b935178c82ede44119501cb287f81-1] regionserver.HStore(310): Store=d14b935178c82ede44119501cb287f81/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:31,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(602): Cleaned up old failed split transaction detritus: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.splits 2023-02-09 20:54:31,347 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:31,348 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened d14b935178c82ede44119501cb287f81; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68441316, jitterRate=0.019855082035064697}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:31,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for d14b935178c82ede44119501cb287f81: Running coprocessor pre-open hook at 1675976071282Writing region info on filesystem at 1675976071282Initializing all the Stores at 1675976071283 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976071283Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976071283Cleaning up temporary data from old regions at 1675976071345 (+62 ms)Cleaning up detritus from prior splits at 1675976071346 (+1 ms)Running coprocessor post-open hooks at 1675976071349 (+3 ms)Region opened successfully at 1675976071349 2023-02-09 20:54:31,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81., pid=48, masterSystemTime=1675976071278 2023-02-09 20:54:31,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:31,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:31,352 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:31,352 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:31,352 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-09 20:54:31,352 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-09 20:54:31,353 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): d14b935178c82ede44119501cb287f81/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:31,353 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): d14b935178c82ede44119501cb287f81/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:31,353 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of d14b935178c82ede44119501cb287f81/cf1 in testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,353 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of d14b935178c82ede44119501cb287f81/cf2 in testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,353 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp, totalSize=34.9 K 2023-02-09 20:54:31,353 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp, totalSize=34.9 K 2023-02-09 20:54:31,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,354 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 713fa478ca0246b7bc54a7cd22aae57a, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976064235 2023-02-09 20:54:31,354 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:31,354 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting f87245d60079446198f87b25997abe7f, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976064235 2023-02-09 20:54:31,354 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976071354"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976071354"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976071354"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976071354"}]},"ts":"1675976071354"} 2023-02-09 20:54:31,354 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 8e6231a0e2744cca865357497f3b485f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976064416 2023-02-09 20:54:31,355 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 28be57de72d74a03b774b3d774cd61b5, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976064416 2023-02-09 20:54:31,355 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 8108a083589f469bbaad0f3398c49244, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976064606 2023-02-09 20:54:31,355 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting b23cea0406f6447fa387b2a244101c66, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976064606 2023-02-09 20:54:31,355 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting f3c7ee99e7e847a5aeb8d6317609942b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976064799 2023-02-09 20:54:31,355 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 46f09f90f4094bb5a32508da572a9423, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976064799 2023-02-09 20:54:31,356 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 46c1ebbdead04aa7897e535bf1e17df6, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976064982 2023-02-09 20:54:31,356 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 7dc3695533a943f0bf00c5c86b493997, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976064982 2023-02-09 20:54:31,356 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting d091b7dbc3814593874eb7cce9f2c03a, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976065165 2023-02-09 20:54:31,356 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting abd8fc686ac642e8822f7e5bd3a4d796, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976065165 2023-02-09 20:54:31,357 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting f4ca3de332b74955878a1ddee826ca31, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976065348 2023-02-09 20:54:31,357 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 12a72bff98c2492f9b1333dac479d273, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976065348 2023-02-09 20:54:31,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-02-09 20:54:31,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 in 230 msec 2023-02-09 20:54:31,360 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN in 1.5560 sec 2023-02-09 20:54:31,375 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(335): procId=44 exception: org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested 2023-02-09 20:54:31,380 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): d14b935178c82ede44119501cb287f81#cf2#compaction#50 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:31,380 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): d14b935178c82ede44119501cb287f81#cf1#compaction#51 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:31,380 DEBUG [Listener at localhost/45551] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRollbackAndDoubleExecution get assigned. Timeout = 1500ms 2023-02-09 20:54:31,382 INFO [Listener at localhost/45551] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-02-09 20:54:31,389 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(3484): All regions for table testRollbackAndDoubleExecution assigned to meta. Checking AM states. 2023-02-09 20:54:31,389 INFO [Listener at localhost/45551] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-02-09 20:54:31,389 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(3504): All regions for table testRollbackAndDoubleExecution assigned. 2023-02-09 20:54:31,402 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf1/2c9207bdc71942b88f467916187fab3c as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/2c9207bdc71942b88f467916187fab3c 2023-02-09 20:54:31,408 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/.tmp/cf2/92d38c3d8eaf434e8f1ba02473a25b26 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/92d38c3d8eaf434e8f1ba02473a25b26 2023-02-09 20:54:31,411 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d14b935178c82ede44119501cb287f81/cf1 of d14b935178c82ede44119501cb287f81 into 2c9207bdc71942b88f467916187fab3c(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:31,412 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:31,412 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81., storeName=d14b935178c82ede44119501cb287f81/cf1, priority=9, startTime=1675976071350; duration=0sec 2023-02-09 20:54:31,412 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:31,418 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d14b935178c82ede44119501cb287f81/cf2 of d14b935178c82ede44119501cb287f81 into 92d38c3d8eaf434e8f1ba02473a25b26(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:31,419 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for d14b935178c82ede44119501cb287f81: 2023-02-09 20:54:31,419 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81., storeName=d14b935178c82ede44119501cb287f81/cf2, priority=9, startTime=1675976071351; duration=0sec 2023-02-09 20:54:31,419 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:31,430 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:31,432 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testRollbackAndDoubleExecution 2023-02-09 20:54:31,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testRollbackAndDoubleExecution 2023-02-09 20:54:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-09 20:54:31,437 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976071437"}]},"ts":"1675976071437"} 2023-02-09 20:54:31,439 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLING in hbase:meta 2023-02-09 20:54:31,441 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackAndDoubleExecution to state=DISABLING 2023-02-09 20:54:31,442 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN}] 2023-02-09 20:54:31,443 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN 2023-02-09 20:54:31,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:31,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976071444"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976071444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976071444"}]},"ts":"1675976071444"} 2023-02-09 20:54:31,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:31,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-09 20:54:31,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing d14b935178c82ede44119501cb287f81, disabling compactions & flushes 2023-02-09 20:54:31,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. after waiting 0 ms 2023-02-09 20:54:31,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,598 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273] to archive 2023-02-09 20:54:31,599 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:31,601 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/f87245d60079446198f87b25997abe7f 2023-02-09 20:54:31,602 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/28be57de72d74a03b774b3d774cd61b5 2023-02-09 20:54:31,603 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/b23cea0406f6447fa387b2a244101c66 2023-02-09 20:54:31,604 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/46f09f90f4094bb5a32508da572a9423 2023-02-09 20:54:31,605 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/7dc3695533a943f0bf00c5c86b493997 2023-02-09 20:54:31,606 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/abd8fc686ac642e8822f7e5bd3a4d796 2023-02-09 20:54:31,608 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/12a72bff98c2492f9b1333dac479d273 2023-02-09 20:54:31,612 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31] to archive 2023-02-09 20:54:31,612 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:31,614 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/713fa478ca0246b7bc54a7cd22aae57a 2023-02-09 20:54:31,615 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8e6231a0e2744cca865357497f3b485f 2023-02-09 20:54:31,616 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/8108a083589f469bbaad0f3398c49244 2023-02-09 20:54:31,617 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f3c7ee99e7e847a5aeb8d6317609942b 2023-02-09 20:54:31,618 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/46c1ebbdead04aa7897e535bf1e17df6 2023-02-09 20:54:31,619 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/d091b7dbc3814593874eb7cce9f2c03a 2023-02-09 20:54:31,620 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/f4ca3de332b74955878a1ddee826ca31 2023-02-09 20:54:31,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:54:31,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:31,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81. 2023-02-09 20:54:31,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for d14b935178c82ede44119501cb287f81: Waiting for close lock at 1675976071597Running coprocessor pre-close hooks at 1675976071597Disabling compacts and flushes for region at 1675976071597Disabling writes for close at 1675976071597Writing region close event to WAL at 1675976071621 (+24 ms)Running coprocessor post-close hooks at 1675976071624 (+3 ms)Closed at 1675976071624 2023-02-09 20:54:31,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,626 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=d14b935178c82ede44119501cb287f81, regionState=CLOSED 2023-02-09 20:54:31,626 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976071626"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976071626"}]},"ts":"1675976071626"} 2023-02-09 20:54:31,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-02-09 20:54:31,629 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; CloseRegionProcedure d14b935178c82ede44119501cb287f81, server=jenkins-hbase9.apache.org,43365,1675976048828 in 183 msec 2023-02-09 20:54:31,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-02-09 20:54:31,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, UNASSIGN in 187 msec 2023-02-09 20:54:31,631 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976071631"}]},"ts":"1675976071631"} 2023-02-09 20:54:31,632 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLED in hbase:meta 2023-02-09 20:54:31,634 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackAndDoubleExecution to state=DISABLED 2023-02-09 20:54:31,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution in 201 msec 2023-02-09 20:54:31,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-09 20:54:31,739 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackAndDoubleExecution, procId: 49 completed 2023-02-09 20:54:31,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testRollbackAndDoubleExecution 2023-02-09 20:54:31,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,743 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,743 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=52, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-02-09 20:54:31,747 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,749 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits] 2023-02-09 20:54:31,752 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/2c9207bdc71942b88f467916187fab3c to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf1/2c9207bdc71942b88f467916187fab3c 2023-02-09 20:54:31,754 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/92d38c3d8eaf434e8f1ba02473a25b26 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/cf2/92d38c3d8eaf434e8f1ba02473a25b26 2023-02-09 20:54:31,756 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81/recovered.edits/90.seqid 2023-02-09 20:54:31,757 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackAndDoubleExecution/d14b935178c82ede44119501cb287f81 2023-02-09 20:54:31,757 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-02-09 20:54:31,760 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=52, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,762 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackAndDoubleExecution from hbase:meta 2023-02-09 20:54:31,763 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackAndDoubleExecution' descriptor. 2023-02-09 20:54:31,764 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=52, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,764 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackAndDoubleExecution' from region states. 2023-02-09 20:54:31,765 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976071764"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:31,766 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-02-09 20:54:31,766 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d14b935178c82ede44119501cb287f81, NAME => 'testRollbackAndDoubleExecution,,1675976063730.d14b935178c82ede44119501cb287f81.', STARTKEY => '', ENDKEY => ''}] 2023-02-09 20:54:31,766 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackAndDoubleExecution' as deleted. 2023-02-09 20:54:31,766 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976071766"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:31,768 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackAndDoubleExecution state from META 2023-02-09 20:54:31,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=52, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:54:31,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution in 29 msec 2023-02-09 20:54:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-02-09 20:54:31,845 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackAndDoubleExecution, procId: 52 completed 2023-02-09 20:54:31,859 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=403 (was 403), OpenFileDescriptor=847 (was 865), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=215 (was 235), ProcessCount=168 (was 168), AvailableMemoryMB=10797 (was 10822) 2023-02-09 20:54:31,873 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=403, OpenFileDescriptor=847, MaxFileDescriptor=60000, SystemLoadAverage=215, ProcessCount=168, AvailableMemoryMB=10797 2023-02-09 20:54:31,873 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:31,876 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:31,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:31,878 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:31,879 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:31,880 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:31,881 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:31,882 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea empty. 2023-02-09 20:54:31,882 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:31,882 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-02-09 20:54:31,893 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:31,894 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(7671): creating {ENCODED => 8a3ca7874d45ec640c0690ef490727ea, NAME => 'testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionNoStoreFile', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:31,903 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:31,903 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1603): Closing 8a3ca7874d45ec640c0690ef490727ea, disabling compactions & flushes 2023-02-09 20:54:31,903 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:31,903 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:31,904 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. after waiting 0 ms 2023-02-09 20:54:31,904 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:31,904 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:31,904 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1557): Region close journal for 8a3ca7874d45ec640c0690ef490727ea: Waiting for close lock at 1675976071903Disabling compacts and flushes for region at 1675976071903Disabling writes for close at 1675976071904 (+1 ms)Writing region close event to WAL at 1675976071904Closed at 1675976071904 2023-02-09 20:54:31,906 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:31,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976071907"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976071907"}]},"ts":"1675976071907"} 2023-02-09 20:54:31,909 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:31,910 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:31,910 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976071910"}]},"ts":"1675976071910"} 2023-02-09 20:54:31,911 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLING in hbase:meta 2023-02-09 20:54:31,915 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:31,916 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:31,916 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:31,916 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:31,916 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:31,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, ASSIGN}] 2023-02-09 20:54:31,918 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, ASSIGN 2023-02-09 20:54:31,918 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:32,069 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:32,070 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=8a3ca7874d45ec640c0690ef490727ea, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,070 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976072070"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976072070"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976072070"}]},"ts":"1675976072070"} 2023-02-09 20:54:32,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE; OpenRegionProcedure 8a3ca7874d45ec640c0690ef490727ea, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:32,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8a3ca7874d45ec640c0690ef490727ea, NAME => 'testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:32,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:32,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:32,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,231 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,233 DEBUG [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf1 2023-02-09 20:54:32,233 DEBUG [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf1 2023-02-09 20:54:32,233 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8a3ca7874d45ec640c0690ef490727ea columnFamilyName cf1 2023-02-09 20:54:32,234 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] regionserver.HStore(310): Store=8a3ca7874d45ec640c0690ef490727ea/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,234 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,236 DEBUG [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf2 2023-02-09 20:54:32,236 DEBUG [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf2 2023-02-09 20:54:32,236 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8a3ca7874d45ec640c0690ef490727ea columnFamilyName cf2 2023-02-09 20:54:32,237 INFO [StoreOpener-8a3ca7874d45ec640c0690ef490727ea-1] regionserver.HStore(310): Store=8a3ca7874d45ec640c0690ef490727ea/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:32,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,245 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:32,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 8a3ca7874d45ec640c0690ef490727ea; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74200071, jitterRate=0.10566721856594086}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:32,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 8a3ca7874d45ec640c0690ef490727ea: Running coprocessor pre-open hook at 1675976072230Writing region info on filesystem at 1675976072230Initializing all the Stores at 1675976072231 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072231Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072231Cleaning up temporary data from old regions at 1675976072239 (+8 ms)Cleaning up detritus from prior splits at 1675976072240 (+1 ms)Running coprocessor post-open hooks at 1675976072246 (+6 ms)Region opened successfully at 1675976072246 2023-02-09 20:54:32,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea., pid=55, masterSystemTime=1675976072223 2023-02-09 20:54:32,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,249 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=8a3ca7874d45ec640c0690ef490727ea, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,249 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976072248"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976072248"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976072248"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976072248"}]},"ts":"1675976072248"} 2023-02-09 20:54:32,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-02-09 20:54:32,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; OpenRegionProcedure 8a3ca7874d45ec640c0690ef490727ea, server=jenkins-hbase9.apache.org,36111,1675976048792 in 178 msec 2023-02-09 20:54:32,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=54, resume processing ppid=53 2023-02-09 20:54:32,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=53, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, ASSIGN in 337 msec 2023-02-09 20:54:32,255 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:32,255 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976072255"}]},"ts":"1675976072255"} 2023-02-09 20:54:32,256 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLED in hbase:meta 2023-02-09 20:54:32,258 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:32,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile in 381 msec 2023-02-09 20:54:32,379 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,380 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,381 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,381 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,381 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,383 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, UNASSIGN}] 2023-02-09 20:54:32,384 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, UNASSIGN 2023-02-09 20:54:32,384 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=8a3ca7874d45ec640c0690ef490727ea, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,384 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976072384"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976072384"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976072384"}]},"ts":"1675976072384"} 2023-02-09 20:54:32,385 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure 8a3ca7874d45ec640c0690ef490727ea, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:32,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 8a3ca7874d45ec640c0690ef490727ea, disabling compactions & flushes 2023-02-09 20:54:32,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. after waiting 0 ms 2023-02-09 20:54:32,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-09 20:54:32,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:32,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea. 2023-02-09 20:54:32,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 8a3ca7874d45ec640c0690ef490727ea: Waiting for close lock at 1675976072536Running coprocessor pre-close hooks at 1675976072536Disabling compacts and flushes for region at 1675976072536Disabling writes for close at 1675976072537 (+1 ms)Writing region close event to WAL at 1675976072538 (+1 ms)Running coprocessor post-close hooks at 1675976072541 (+3 ms)Closed at 1675976072541 2023-02-09 20:54:32,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:32,543 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=8a3ca7874d45ec640c0690ef490727ea, regionState=CLOSED 2023-02-09 20:54:32,543 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976072543"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976072543"}]},"ts":"1675976072543"} 2023-02-09 20:54:32,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-02-09 20:54:32,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure 8a3ca7874d45ec640c0690ef490727ea, server=jenkins-hbase9.apache.org,36111,1675976048792 in 160 msec 2023-02-09 20:54:32,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=56 2023-02-09 20:54:32,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=8a3ca7874d45ec640c0690ef490727ea, UNASSIGN in 163 msec 2023-02-09 20:54:32,571 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-02-09 20:54:32,573 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-02-09 20:54:32,574 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1675976072574"},{"qualifier":"splitA","vlen":67,"tag":[],"timestamp":"1675976072574"},{"qualifier":"splitB","vlen":67,"tag":[],"timestamp":"1675976072574"}]},"ts":"1675976072574"} 2023-02-09 20:54:32,574 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976072574"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976072574"}]},"ts":"1675976072574"} 2023-02-09 20:54:32,574 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976072574"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976072574"}]},"ts":"1675976072574"} 2023-02-09 20:54:32,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, ASSIGN}, {pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, ASSIGN}] 2023-02-09 20:54:32,578 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, ASSIGN 2023-02-09 20:54:32,578 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:32,579 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, ASSIGN 2023-02-09 20:54:32,580 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:32,729 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:54:32,730 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5e2dbb380e7a15a98ea33055bf2e466c, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,731 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072730"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976072730"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976072730"}]},"ts":"1675976072730"} 2023-02-09 20:54:32,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 5e2dbb380e7a15a98ea33055bf2e466c, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:32,735 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=41c1b8256357c0b55c682056898fc851, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,735 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072735"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976072735"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976072735"}]},"ts":"1675976072735"} 2023-02-09 20:54:32,736 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=62, ppid=60, state=RUNNABLE; OpenRegionProcedure 41c1b8256357c0b55c682056898fc851, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:32,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:32,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 5e2dbb380e7a15a98ea33055bf2e466c, NAME => 'testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.', STARTKEY => '', ENDKEY => '41'} 2023-02-09 20:54:32,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:32,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:32,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,889 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,890 DEBUG [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf1 2023-02-09 20:54:32,890 DEBUG [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf1 2023-02-09 20:54:32,891 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5e2dbb380e7a15a98ea33055bf2e466c columnFamilyName cf1 2023-02-09 20:54:32,891 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] regionserver.HStore(310): Store=5e2dbb380e7a15a98ea33055bf2e466c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,891 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,893 DEBUG [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf2 2023-02-09 20:54:32,893 DEBUG [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf2 2023-02-09 20:54:32,893 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5e2dbb380e7a15a98ea33055bf2e466c columnFamilyName cf2 2023-02-09 20:54:32,893 INFO [StoreOpener-5e2dbb380e7a15a98ea33055bf2e466c-1] regionserver.HStore(310): Store=5e2dbb380e7a15a98ea33055bf2e466c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:32,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:32,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 5e2dbb380e7a15a98ea33055bf2e466c; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63525725, jitterRate=-0.0533929318189621}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:32,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 5e2dbb380e7a15a98ea33055bf2e466c: Running coprocessor pre-open hook at 1675976072888Writing region info on filesystem at 1675976072888Initializing all the Stores at 1675976072889 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072889Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072889Cleaning up temporary data from old regions at 1675976072896 (+7 ms)Cleaning up detritus from prior splits at 1675976072896Running coprocessor post-open hooks at 1675976072898 (+2 ms)Region opened successfully at 1675976072898 2023-02-09 20:54:32,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c., pid=61, masterSystemTime=1675976072884 2023-02-09 20:54:32,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:32,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:32,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:32,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 41c1b8256357c0b55c682056898fc851, NAME => 'testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.', STARTKEY => '41', ENDKEY => ''} 2023-02-09 20:54:32,901 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5e2dbb380e7a15a98ea33055bf2e466c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:32,901 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072901"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976072901"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976072901"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976072901"}]},"ts":"1675976072901"} 2023-02-09 20:54:32,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:32,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,902 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,904 DEBUG [StoreOpener-41c1b8256357c0b55c682056898fc851-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf1 2023-02-09 20:54:32,904 DEBUG [StoreOpener-41c1b8256357c0b55c682056898fc851-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf1 2023-02-09 20:54:32,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-02-09 20:54:32,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 5e2dbb380e7a15a98ea33055bf2e466c, server=jenkins-hbase9.apache.org,36111,1675976048792 in 170 msec 2023-02-09 20:54:32,904 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c1b8256357c0b55c682056898fc851 columnFamilyName cf1 2023-02-09 20:54:32,905 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] regionserver.HStore(310): Store=41c1b8256357c0b55c682056898fc851/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,905 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, ASSIGN in 327 msec 2023-02-09 20:54:32,906 DEBUG [StoreOpener-41c1b8256357c0b55c682056898fc851-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf2 2023-02-09 20:54:32,906 DEBUG [StoreOpener-41c1b8256357c0b55c682056898fc851-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf2 2023-02-09 20:54:32,906 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c1b8256357c0b55c682056898fc851 columnFamilyName cf2 2023-02-09 20:54:32,907 INFO [StoreOpener-41c1b8256357c0b55c682056898fc851-1] regionserver.HStore(310): Store=41c1b8256357c0b55c682056898fc851/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:32,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:32,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:32,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 41c1b8256357c0b55c682056898fc851; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71854710, jitterRate=0.07071861624717712}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:32,912 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 41c1b8256357c0b55c682056898fc851: Running coprocessor pre-open hook at 1675976072901Writing region info on filesystem at 1675976072901Initializing all the Stores at 1675976072902 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072902Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976072902Cleaning up temporary data from old regions at 1675976072909 (+7 ms)Cleaning up detritus from prior splits at 1675976072909Running coprocessor post-open hooks at 1675976072912 (+3 ms)Region opened successfully at 1675976072912 2023-02-09 20:54:32,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851., pid=62, masterSystemTime=1675976072884 2023-02-09 20:54:32,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:32,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:32,914 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=41c1b8256357c0b55c682056898fc851, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:32,914 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976072914"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976072914"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976072914"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976072914"}]},"ts":"1675976072914"} 2023-02-09 20:54:32,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=62, resume processing ppid=60 2023-02-09 20:54:32,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, ppid=60, state=SUCCESS; OpenRegionProcedure 41c1b8256357c0b55c682056898fc851, server=jenkins-hbase9.apache.org,36111,1675976048792 in 179 msec 2023-02-09 20:54:32,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=56 2023-02-09 20:54:32,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, ASSIGN in 340 msec 2023-02-09 20:54:32,920 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 in 540 msec 2023-02-09 20:54:33,135 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:54:33,137 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:33,139 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-09 20:54:33,142 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976073142"}]},"ts":"1675976073142"} 2023-02-09 20:54:33,143 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLING in hbase:meta 2023-02-09 20:54:33,146 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionNoStoreFile to state=DISABLING 2023-02-09 20:54:33,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, UNASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, UNASSIGN}] 2023-02-09 20:54:33,147 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, UNASSIGN 2023-02-09 20:54:33,148 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=41c1b8256357c0b55c682056898fc851, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:33,148 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976073148"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976073148"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976073148"}]},"ts":"1675976073148"} 2023-02-09 20:54:33,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; CloseRegionProcedure 41c1b8256357c0b55c682056898fc851, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:33,151 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, UNASSIGN 2023-02-09 20:54:33,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=5e2dbb380e7a15a98ea33055bf2e466c, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:33,152 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976073152"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976073152"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976073152"}]},"ts":"1675976073152"} 2023-02-09 20:54:33,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE; CloseRegionProcedure 5e2dbb380e7a15a98ea33055bf2e466c, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:33,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-09 20:54:33,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:33,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 41c1b8256357c0b55c682056898fc851, disabling compactions & flushes 2023-02-09 20:54:33,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:33,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:33,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. after waiting 0 ms 2023-02-09 20:54:33,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:33,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-09 20:54:33,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:33,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851. 2023-02-09 20:54:33,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 41c1b8256357c0b55c682056898fc851: Waiting for close lock at 1675976073301Running coprocessor pre-close hooks at 1675976073301Disabling compacts and flushes for region at 1675976073301Disabling writes for close at 1675976073302 (+1 ms)Writing region close event to WAL at 1675976073302Running coprocessor post-close hooks at 1675976073306 (+4 ms)Closed at 1675976073306 2023-02-09 20:54:33,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:33,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:33,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 5e2dbb380e7a15a98ea33055bf2e466c, disabling compactions & flushes 2023-02-09 20:54:33,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:33,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:33,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. after waiting 0 ms 2023-02-09 20:54:33,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:33,309 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=41c1b8256357c0b55c682056898fc851, regionState=CLOSED 2023-02-09 20:54:33,309 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976073309"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976073309"}]},"ts":"1675976073309"} 2023-02-09 20:54:33,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=65 2023-02-09 20:54:33,312 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=65, state=SUCCESS; CloseRegionProcedure 41c1b8256357c0b55c682056898fc851, server=jenkins-hbase9.apache.org,36111,1675976048792 in 161 msec 2023-02-09 20:54:33,313 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=41c1b8256357c0b55c682056898fc851, UNASSIGN in 166 msec 2023-02-09 20:54:33,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-09 20:54:33,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:33,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c. 2023-02-09 20:54:33,313 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 5e2dbb380e7a15a98ea33055bf2e466c: Waiting for close lock at 1675976073308Running coprocessor pre-close hooks at 1675976073308Disabling compacts and flushes for region at 1675976073308Disabling writes for close at 1675976073309 (+1 ms)Writing region close event to WAL at 1675976073309Running coprocessor post-close hooks at 1675976073313 (+4 ms)Closed at 1675976073313 2023-02-09 20:54:33,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:33,315 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=5e2dbb380e7a15a98ea33055bf2e466c, regionState=CLOSED 2023-02-09 20:54:33,315 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976073315"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976073315"}]},"ts":"1675976073315"} 2023-02-09 20:54:33,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=64 2023-02-09 20:54:33,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=64, state=SUCCESS; CloseRegionProcedure 5e2dbb380e7a15a98ea33055bf2e466c, server=jenkins-hbase9.apache.org,36111,1675976048792 in 163 msec 2023-02-09 20:54:33,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-02-09 20:54:33,319 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=5e2dbb380e7a15a98ea33055bf2e466c, UNASSIGN in 172 msec 2023-02-09 20:54:33,320 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976073320"}]},"ts":"1675976073320"} 2023-02-09 20:54:33,321 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLED in hbase:meta 2023-02-09 20:54:33,322 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionNoStoreFile to state=DISABLED 2023-02-09 20:54:33,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile in 184 msec 2023-02-09 20:54:33,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-09 20:54:33,444 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 63 completed 2023-02-09 20:54:33,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,448 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,448 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=68, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-02-09 20:54:33,453 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:33,453 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:33,453 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:33,455 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/recovered.edits] 2023-02-09 20:54:33,455 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/recovered.edits] 2023-02-09 20:54:33,455 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/recovered.edits] 2023-02-09 20:54:33,463 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/recovered.edits/7.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c/recovered.edits/7.seqid 2023-02-09 20:54:33,463 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/recovered.edits/4.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea/recovered.edits/4.seqid 2023-02-09 20:54:33,464 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/5e2dbb380e7a15a98ea33055bf2e466c 2023-02-09 20:54:33,464 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/8a3ca7874d45ec640c0690ef490727ea 2023-02-09 20:54:33,464 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/recovered.edits/7.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851/recovered.edits/7.seqid 2023-02-09 20:54:33,464 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionNoStoreFile/41c1b8256357c0b55c682056898fc851 2023-02-09 20:54:33,464 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-02-09 20:54:33,467 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=68, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,468 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionNoStoreFile from hbase:meta 2023-02-09 20:54:33,470 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionNoStoreFile' descriptor. 2023-02-09 20:54:33,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=68, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionNoStoreFile' from region states. 2023-02-09 20:54:33,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976073471"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:33,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976073471"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:33,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976073471"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:33,473 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:54:33,473 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8a3ca7874d45ec640c0690ef490727ea, NAME => 'testSplitTableRegionNoStoreFile,,1675976071877.8a3ca7874d45ec640c0690ef490727ea.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 5e2dbb380e7a15a98ea33055bf2e466c, NAME => 'testSplitTableRegionNoStoreFile,,1675976072379.5e2dbb380e7a15a98ea33055bf2e466c.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 41c1b8256357c0b55c682056898fc851, NAME => 'testSplitTableRegionNoStoreFile,41,1675976072379.41c1b8256357c0b55c682056898fc851.', STARTKEY => '41', ENDKEY => ''}] 2023-02-09 20:54:33,473 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionNoStoreFile' as deleted. 2023-02-09 20:54:33,473 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976073473"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:33,474 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionNoStoreFile state from META 2023-02-09 20:54:33,476 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=68, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:54:33,477 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile in 31 msec 2023-02-09 20:54:33,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-02-09 20:54:33,550 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 68 completed 2023-02-09 20:54:33,564 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=407 (was 403) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_13972287_15 at /127.0.0.1:48888 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_13972287_15 at /127.0.0.1:57112 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=847 (was 847), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=215 (was 215), ProcessCount=168 (was 168), AvailableMemoryMB=10796 (was 10797) 2023-02-09 20:54:33,578 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=407, OpenFileDescriptor=847, MaxFileDescriptor=60000, SystemLoadAverage=215, ProcessCount=168, AvailableMemoryMB=10795 2023-02-09 20:54:33,578 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:33,581 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:33,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:33,583 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:33,585 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:33,585 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:33,587 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,587 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 empty. 2023-02-09 20:54:33,588 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,588 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-02-09 20:54:33,602 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:33,604 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => f914af6ac487ff13c2af3eebff56a1b1, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionDeletedRowsDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1603): Closing f914af6ac487ff13c2af3eebff56a1b1, disabling compactions & flushes 2023-02-09 20:54:33,616 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. after waiting 0 ms 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,616 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,616 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1557): Region close journal for f914af6ac487ff13c2af3eebff56a1b1: Waiting for close lock at 1675976073616Disabling compacts and flushes for region at 1675976073616Disabling writes for close at 1675976073616Writing region close event to WAL at 1675976073616Closed at 1675976073616 2023-02-09 20:54:33,619 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:33,619 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976073619"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976073619"}]},"ts":"1675976073619"} 2023-02-09 20:54:33,621 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:33,621 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:33,622 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976073621"}]},"ts":"1675976073621"} 2023-02-09 20:54:33,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLING in hbase:meta 2023-02-09 20:54:33,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:33,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:33,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:33,627 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:33,627 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:33,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, ASSIGN}] 2023-02-09 20:54:33,628 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, ASSIGN 2023-02-09 20:54:33,629 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:33,779 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:33,780 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=f914af6ac487ff13c2af3eebff56a1b1, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:33,780 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976073780"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976073780"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976073780"}]},"ts":"1675976073780"} 2023-02-09 20:54:33,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE; OpenRegionProcedure f914af6ac487ff13c2af3eebff56a1b1, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:33,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f914af6ac487ff13c2af3eebff56a1b1, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:33,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:33,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:33,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,938 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,940 DEBUG [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1 2023-02-09 20:54:33,940 DEBUG [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1 2023-02-09 20:54:33,940 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f914af6ac487ff13c2af3eebff56a1b1 columnFamilyName cf1 2023-02-09 20:54:33,940 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] regionserver.HStore(310): Store=f914af6ac487ff13c2af3eebff56a1b1/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:33,941 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,942 DEBUG [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2 2023-02-09 20:54:33,942 DEBUG [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2 2023-02-09 20:54:33,942 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f914af6ac487ff13c2af3eebff56a1b1 columnFamilyName cf2 2023-02-09 20:54:33,943 INFO [StoreOpener-f914af6ac487ff13c2af3eebff56a1b1-1] regionserver.HStore(310): Store=f914af6ac487ff13c2af3eebff56a1b1/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:33,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:33,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:33,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:33,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened f914af6ac487ff13c2af3eebff56a1b1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68117479, jitterRate=0.015029534697532654}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:33,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for f914af6ac487ff13c2af3eebff56a1b1: Running coprocessor pre-open hook at 1675976073937Writing region info on filesystem at 1675976073937Initializing all the Stores at 1675976073938 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976073938Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976073938Cleaning up temporary data from old regions at 1675976073945 (+7 ms)Cleaning up detritus from prior splits at 1675976073945Running coprocessor post-open hooks at 1675976073951 (+6 ms)Region opened successfully at 1675976073951 2023-02-09 20:54:33,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1., pid=71, masterSystemTime=1675976073933 2023-02-09 20:54:33,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:33,956 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=f914af6ac487ff13c2af3eebff56a1b1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:33,957 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976073956"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976073956"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976073956"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976073956"}]},"ts":"1675976073956"} 2023-02-09 20:54:33,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=70 2023-02-09 20:54:33,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=70, state=SUCCESS; OpenRegionProcedure f914af6ac487ff13c2af3eebff56a1b1, server=jenkins-hbase9.apache.org,36111,1675976048792 in 177 msec 2023-02-09 20:54:33,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=69 2023-02-09 20:54:33,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=69, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, ASSIGN in 333 msec 2023-02-09 20:54:33,962 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:33,962 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976073962"}]},"ts":"1675976073962"} 2023-02-09 20:54:33,963 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLED in hbase:meta 2023-02-09 20:54:33,966 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:33,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 384 msec 2023-02-09 20:54:34,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:34,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,092 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,093 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,093 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:34,093 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:34,093 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,093 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,095 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,095 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,095 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,095 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,096 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,096 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:34,097 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:34,097 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,097 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:34,098 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:54:34,099 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,099 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,099 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,100 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,100 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,100 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,117 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/55af41f66c7f464ca7ecac7c074da705 2023-02-09 20:54:34,130 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/d2d38e5493664d76b20f718170beb5dc 2023-02-09 20:54:34,134 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/55af41f66c7f464ca7ecac7c074da705 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705 2023-02-09 20:54:34,138 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:34,139 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/d2d38e5493664d76b20f718170beb5dc as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc 2023-02-09 20:54:34,142 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:54:34,143 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 45ms, sequenceid=6, compaction requested=false 2023-02-09 20:54:34,143 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,144 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:34,144 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,144 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:34,144 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:34,144 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,144 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,144 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,146 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,146 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,146 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,146 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:34,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2016e0b2[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:34,149 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:34,149 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,152 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,153 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,153 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,155 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,156 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,156 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,156 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,156 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:34,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:34,159 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:34,159 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:34,159 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:34,159 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,159 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,162 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,163 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,164 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,164 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,166 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,166 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,166 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,173 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,174 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,274 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:34,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:34,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:34,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:34,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,291 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:34,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:34,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,294 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,294 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,294 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,294 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,295 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,295 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:34,296 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,297 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:34,296 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,297 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:34,297 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,298 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,299 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,299 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,299 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,299 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,299 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,317 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/39985665b359468fbdd6d457d3bdbdac 2023-02-09 20:54:34,339 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/f9e12424936a4c939783032b70326d7e 2023-02-09 20:54:34,344 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/39985665b359468fbdd6d457d3bdbdac as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac 2023-02-09 20:54:34,348 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:34,349 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/f9e12424936a4c939783032b70326d7e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e 2023-02-09 20:54:34,353 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:54:34,354 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 57ms, sequenceid=19, compaction requested=false 2023-02-09 20:54:34,354 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:34,354 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,354 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:34,354 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:34,354 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,354 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,354 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,356 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:34,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a2a1600[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:34,358 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:34,358 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,361 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:34,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,361 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:34,361 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:34,362 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,362 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,362 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,363 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,363 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:34,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:34,366 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:34,366 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:34,366 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:34,366 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,366 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,368 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,369 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,369 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,371 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,371 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,372 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,375 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:34,375 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,376 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,377 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,377 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,476 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:34,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:34,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:34,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:34,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,491 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:34,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:34,491 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,492 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,493 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,493 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,493 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,493 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:34,495 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:34,495 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,495 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:34,496 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,497 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,509 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/ee51309bb4224e1eb5b06abd333fa570 2023-02-09 20:54:34,521 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/20be9bf7ed3646568a2b2e12754face0 2023-02-09 20:54:34,525 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/ee51309bb4224e1eb5b06abd333fa570 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570 2023-02-09 20:54:34,529 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:34,530 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/20be9bf7ed3646568a2b2e12754face0 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0 2023-02-09 20:54:34,533 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:54:34,534 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 38ms, sequenceid=32, compaction requested=true 2023-02-09 20:54:34,534 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:34,534 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:34,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:34,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,543 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,543 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,544 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:34,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a6b6b5[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:34,545 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:34,546 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:34,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,547 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,547 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,548 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,548 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,549 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,549 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,551 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,551 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,551 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:34,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:34,553 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:34,553 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:34,553 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:34,553 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,553 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,555 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,556 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,556 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,558 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,558 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,558 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,561 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,561 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:34,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,562 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,662 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:34,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:34,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:34,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:34,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,677 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,678 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,678 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:34,678 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:34,678 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,678 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,679 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,680 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,680 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,681 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,681 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,682 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,682 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,682 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:34,682 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:34,684 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,685 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,685 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,685 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,685 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,685 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,691 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/eb99041bef0d40caadb25334afcded5b 2023-02-09 20:54:34,705 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/efe729ad4ed549409aae127a40ad112f 2023-02-09 20:54:34,709 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/eb99041bef0d40caadb25334afcded5b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b 2023-02-09 20:54:34,713 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:34,714 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/efe729ad4ed549409aae127a40ad112f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f 2023-02-09 20:54:34,717 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:54:34,718 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 36ms, sequenceid=45, compaction requested=true 2023-02-09 20:54:34,718 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:34,719 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,719 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:34,719 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:34,719 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,719 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,719 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,721 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,721 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,721 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,721 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:34,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2026496[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:34,723 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:34,723 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,724 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,724 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:34,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,724 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,725 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:34,727 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,727 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,727 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,727 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,728 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:34,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:34,731 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:34,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:34,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:34,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,731 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,735 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,735 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,736 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,736 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,738 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,738 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,738 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,741 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,742 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,842 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:34,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:34,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,861 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:34,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:34,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,861 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,862 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,863 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,863 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,863 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:34,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:34,865 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:34,865 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,865 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:34,865 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,875 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/a37abf412cfe464481967bf740c784c3 2023-02-09 20:54:34,886 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/49b11d79c2fb431290af079dd778da83 2023-02-09 20:54:34,889 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/a37abf412cfe464481967bf740c784c3 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3 2023-02-09 20:54:34,893 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:34,893 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/49b11d79c2fb431290af079dd778da83 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83 2023-02-09 20:54:34,897 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:54:34,898 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 32ms, sequenceid=58, compaction requested=true 2023-02-09 20:54:34,898 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:34,898 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:34,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:34,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:34,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:34,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:34,900 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,901 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,901 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:34,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15861714[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:34,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:34,903 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,904 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:34,904 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,906 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,907 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:34,907 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,907 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:34,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:34,910 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:34,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:34,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:34,910 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,910 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:34,913 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:34,913 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,913 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:34,914 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,915 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,915 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,917 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,917 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,918 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:34,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:34,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:34,921 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,921 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:34,920 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:34,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,921 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:34,921 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:34,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:34,922 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,022 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:35,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:35,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,036 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,036 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,036 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:35,036 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:35,037 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,037 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,040 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,040 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,041 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,041 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:35,042 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:35,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,043 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,043 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,042 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:35,043 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:54:35,045 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,046 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,046 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,046 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,047 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,047 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,052 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/13b7bfe86c384e1893a6aab3f2eb901d 2023-02-09 20:54:35,064 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/170abd2c6fc148199da85887acb915a6 2023-02-09 20:54:35,068 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/13b7bfe86c384e1893a6aab3f2eb901d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d 2023-02-09 20:54:35,072 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:35,073 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/170abd2c6fc148199da85887acb915a6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6 2023-02-09 20:54:35,077 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:54:35,078 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 35ms, sequenceid=71, compaction requested=true 2023-02-09 20:54:35,078 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:35,078 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,078 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:35,078 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:35,078 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,078 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,078 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,080 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,080 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,080 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,080 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:35,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8699b2[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:35,082 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:35,082 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,083 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,083 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:35,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,083 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:35,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,084 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:35,088 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,088 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,088 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,089 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,089 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:35,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:35,092 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:35,092 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:35,092 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:35,092 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,092 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,094 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,094 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,095 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,095 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,101 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:35,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:35,101 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,102 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,202 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:35,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:35,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:35,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:35,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,216 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,216 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,216 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:35,216 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:35,217 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,217 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,219 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,219 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,219 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,219 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:35,221 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:35,221 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,221 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:35,221 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=530 B heapSize=2.33 KB 2023-02-09 20:54:35,222 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,223 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,223 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,223 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,224 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,224 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,231 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:35,235 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:35,242 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:35,246 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:35,246 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/1eda5274095b477093ba8a1a5dc3733c as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:35,250 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:35,250 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c, entries=9, sequenceid=83, filesize=5.1 K 2023-02-09 20:54:35,251 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/e0996e212b8b4f54b155564d3c53b37e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:35,254 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:35,254 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e, entries=9, sequenceid=83, filesize=5.1 K 2023-02-09 20:54:35,255 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~530 B/530, heapSize ~2.30 KB/2352, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 34ms, sequenceid=83, compaction requested=true 2023-02-09 20:54:35,255 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:35,255 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,255 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:35,255 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:35,255 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,255 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,255 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,256 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,256 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,257 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,257 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697385e9[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:35,259 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:35,259 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,261 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,261 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,261 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:35,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,262 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:35,264 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,264 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,264 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,267 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,267 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,267 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,269 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,269 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,269 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,269 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:35,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:35,273 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:35,273 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:35,273 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:35,273 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,273 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:35,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,275 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,276 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,276 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,278 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,278 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,279 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,281 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,282 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,283 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,383 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:35,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:35,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:35,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:35,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,394 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:35,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:35,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,396 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,396 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,396 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,397 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,397 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,397 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,397 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:35,398 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,398 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,398 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,398 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-02-09 20:54:35,400 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,401 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,401 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,401 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,402 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,402 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,408 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:35,436 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:35,444 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:35,447 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:35,448 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:35,451 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:35,451 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6, entries=5, sequenceid=91, filesize=5.0 K 2023-02-09 20:54:35,452 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/5b1a4fb0efa84f7c88035445ebf759ec as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:35,455 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:35,455 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec, entries=5, sequenceid=91, filesize=5.0 K 2023-02-09 20:54:35,456 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 58ms, sequenceid=91, compaction requested=true 2023-02-09 20:54:35,456 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:35,456 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,456 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:35,456 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:35,456 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,456 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,456 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,458 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,458 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,458 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,458 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:35,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c47a85[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:35,459 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:35,460 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:35,461 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:35,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,461 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:35,464 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,464 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,465 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,465 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:35,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:35,467 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:35,467 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:35,467 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:35,467 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,468 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,469 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,470 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,470 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,471 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,472 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,472 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,472 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,478 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,478 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,479 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,480 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,579 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:35,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:54:35,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:54:35,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,589 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,589 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,590 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-02-09 20:54:35,590 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-02-09 20:54:35,590 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,590 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,591 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,591 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,591 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,591 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,592 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,592 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,593 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:54:35,594 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:54:35,594 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,594 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. started... 2023-02-09 20:54:35,594 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-02-09 20:54:35,595 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,595 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,596 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,596 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,596 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,596 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,606 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:35,609 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:35,618 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:35,622 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:35,623 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/026137f74cfd4840a1dbbca1c112358b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:35,628 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:35,628 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b, entries=5, sequenceid=99, filesize=5.0 K 2023-02-09 20:54:35,629 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/d349cf4b521c483d902dd7bd1fb31fea as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:35,632 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:35,633 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea, entries=5, sequenceid=99, filesize=5.0 K 2023-02-09 20:54:35,633 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 39ms, sequenceid=99, compaction requested=true 2023-02-09 20:54:35,634 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for f914af6ac487ff13c2af3eebff56a1b1: 2023-02-09 20:54:35,634 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,634 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:54:35,634 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:54:35,634 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,635 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-02-09 20:54:35,635 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-02-09 20:54:35,636 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,636 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,636 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,636 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:54:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-02-09 20:54:35,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddfec3e[Count = 0] remaining members to acquire global barrier 2023-02-09 20:54:35,638 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-02-09 20:54:35,638 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,641 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:54:35,641 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-02-09 20:54:35,641 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:54:35,642 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-02-09 20:54:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,643 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-02-09 20:54:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:54:35,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:54:35,646 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-02-09 20:54:35,646 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:54:35,646 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:54:35,646 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,646 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,647 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:54:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,648 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:54:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,648 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,650 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,650 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:54:35,651 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,654 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,654 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-02-09 20:54:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:54:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:54:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:54:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:35,655 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:54:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,656 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:35,756 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:54:35,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:54:35,759 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,760 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:35,761 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:35,761 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:35,761 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:35,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, UNASSIGN}] 2023-02-09 20:54:35,763 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, UNASSIGN 2023-02-09 20:54:35,764 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=f914af6ac487ff13c2af3eebff56a1b1, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:35,764 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976075764"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976075764"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976075764"}]},"ts":"1675976075764"} 2023-02-09 20:54:35,765 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE; CloseRegionProcedure f914af6ac487ff13c2af3eebff56a1b1, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:35,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing f914af6ac487ff13c2af3eebff56a1b1, disabling compactions & flushes 2023-02-09 20:54:35,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. after waiting 0 ms 2023-02-09 20:54:35,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing f914af6ac487ff13c2af3eebff56a1b1 2/2 column families, dataSize=50 B heapSize=720 B 2023-02-09 20:54:35,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:35,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:35,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:35,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:35,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf1/d922d8d2f7b24ed38169e021ec403a45 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:35,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:35,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45, entries=1, sequenceid=103, filesize=4.9 K 2023-02-09 20:54:35,947 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.tmp/cf2/3e4e36afdcc743c49295f8a3fdad16ea as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:35,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:35,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea, entries=1, sequenceid=103, filesize=4.9 K 2023-02-09 20:54:35,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 34ms, sequenceid=103, compaction requested=true 2023-02-09 20:54:35,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=1 2023-02-09 20:54:35,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:35,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. 2023-02-09 20:54:35,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for f914af6ac487ff13c2af3eebff56a1b1: Waiting for close lock at 1675976075917Running coprocessor pre-close hooks at 1675976075917Disabling compacts and flushes for region at 1675976075917Disabling writes for close at 1675976075917Obtaining lock to block concurrent updates at 1675976075918 (+1 ms)Preparing flush snapshotting stores in f914af6ac487ff13c2af3eebff56a1b1 at 1675976075918Finished memstore snapshotting testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1., syncing WAL and waiting on mvcc, flushsize=dataSize=50, getHeapSize=688, getOffHeapSize=0, getCellsCount=2 at 1675976075918Flushing stores of testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1. at 1675976075918Flushing f914af6ac487ff13c2af3eebff56a1b1/cf1: creating writer at 1675976075918Flushing f914af6ac487ff13c2af3eebff56a1b1/cf1: appending metadata at 1675976075921 (+3 ms)Flushing f914af6ac487ff13c2af3eebff56a1b1/cf1: closing flushed file at 1675976075921Flushing f914af6ac487ff13c2af3eebff56a1b1/cf2: creating writer at 1675976075932 (+11 ms)Flushing f914af6ac487ff13c2af3eebff56a1b1/cf2: appending metadata at 1675976075933 (+1 ms)Flushing f914af6ac487ff13c2af3eebff56a1b1/cf2: closing flushed file at 1675976075933Flushing f914af6ac487ff13c2af3eebff56a1b1/cf1: reopening flushed file at 1675976075943 (+10 ms)Flushing f914af6ac487ff13c2af3eebff56a1b1/cf2: reopening flushed file at 1675976075948 (+5 ms)Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for f914af6ac487ff13c2af3eebff56a1b1 in 34ms, sequenceid=103, compaction requested=true at 1675976075951 (+3 ms)Writing region close event to WAL at 1675976075952 (+1 ms)Running coprocessor post-close hooks at 1675976075955 (+3 ms)Closed at 1675976075955 2023-02-09 20:54:35,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,957 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=f914af6ac487ff13c2af3eebff56a1b1, regionState=CLOSED 2023-02-09 20:54:35,957 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976075957"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976075957"}]},"ts":"1675976075957"} 2023-02-09 20:54:35,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=73 2023-02-09 20:54:35,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=73, state=SUCCESS; CloseRegionProcedure f914af6ac487ff13c2af3eebff56a1b1, server=jenkins-hbase9.apache.org,36111,1675976048792 in 193 msec 2023-02-09 20:54:35,962 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-02-09 20:54:35,962 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=f914af6ac487ff13c2af3eebff56a1b1, UNASSIGN in 197 msec 2023-02-09 20:54:35,976 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=72 splitting 20 storefiles, region=f914af6ac487ff13c2af3eebff56a1b1, threads=16 2023-02-09 20:54:35,977 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,977 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,977 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,977 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,979 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,979 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,984 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,984 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,984 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,985 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,985 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,987 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,987 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,987 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,991 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,991 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:35,993 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83, top=false 2023-02-09 20:54:35,997 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6, top=false 2023-02-09 20:54:35,997 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f, top=false 2023-02-09 20:54:35,999 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:36,001 INFO [StoreFileSplitter-pool-6] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:36,003 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:36,004 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:36,008 INFO [StoreFileSplitter-pool-2] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:36,009 INFO [StoreFileSplitter-pool-4] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:36,009 INFO [StoreFileSplitter-pool-13] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:36,017 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d, top=false 2023-02-09 20:54:36,026 INFO [StoreFileSplitter-pool-6] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:36,026 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea, top=true 2023-02-09 20:54:36,031 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:36,031 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b, top=true 2023-02-09 20:54:36,032 INFO [StoreFileSplitter-pool-2] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:36,032 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea, top=true 2023-02-09 20:54:36,032 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:36,032 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6, top=true 2023-02-09 20:54:36,035 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,035 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6 for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,035 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83 for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,039 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,039 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,040 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,040 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,040 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,040 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,043 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:36,046 INFO [StoreFileSplitter-pool-3] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:36,046 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45, top=true 2023-02-09 20:54:36,061 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,061 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,061 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,061 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,061 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,061 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,062 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6 for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,062 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,069 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,069 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,075 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,086 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45 for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,087 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,108 INFO [StoreFileSplitter-pool-4] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:36,109 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec, top=true 2023-02-09 20:54:36,109 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3, top=false 2023-02-09 20:54:36,109 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b, top=false 2023-02-09 20:54:36,110 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:36,119 INFO [StoreFileSplitter-pool-13] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:36,156 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,171 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,171 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,172 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3 for child: c61416884e2b4074206df3e8b1ddee75, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,175 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,175 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,179 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,181 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/.splits/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec for child: 68761db75920bcee7891cf83b0ee94c4, parent: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,181 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,199 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,199 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,203 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,204 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,205 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570 for region: f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,205 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=72 split storefiles for region f914af6ac487ff13c2af3eebff56a1b1 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:54:36,247 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-02-09 20:54:36,250 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-02-09 20:54:36,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1675976076251"},{"qualifier":"splitA","vlen":75,"tag":[],"timestamp":"1675976076251"},{"qualifier":"splitB","vlen":75,"tag":[],"timestamp":"1675976076251"}]},"ts":"1675976076251"} 2023-02-09 20:54:36,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076251"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976076251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976076251"}]},"ts":"1675976076251"} 2023-02-09 20:54:36,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076251"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976076251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976076251"}]},"ts":"1675976076251"} 2023-02-09 20:54:36,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, ASSIGN}, {pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, ASSIGN}] 2023-02-09 20:54:36,255 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, ASSIGN 2023-02-09 20:54:36,256 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:36,256 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, ASSIGN 2023-02-09 20:54:36,257 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:54:36,406 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:54:36,408 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=68761db75920bcee7891cf83b0ee94c4, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,408 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076408"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976076408"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976076408"}]},"ts":"1675976076408"} 2023-02-09 20:54:36,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE; OpenRegionProcedure 68761db75920bcee7891cf83b0ee94c4, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:36,412 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=c61416884e2b4074206df3e8b1ddee75, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,412 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076412"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976076412"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976076412"}]},"ts":"1675976076412"} 2023-02-09 20:54:36,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=78, ppid=75, state=RUNNABLE; OpenRegionProcedure c61416884e2b4074206df3e8b1ddee75, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:36,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,564 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 68761db75920bcee7891cf83b0ee94c4, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.', STARTKEY => '60', ENDKEY => ''} 2023-02-09 20:54:36,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:36,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:36,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,566 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,567 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1 2023-02-09 20:54:36,568 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1 2023-02-09 20:54:36,568 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68761db75920bcee7891cf83b0ee94c4 columnFamilyName cf1 2023-02-09 20:54:36,577 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,577 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-top 2023-02-09 20:54:36,581 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-top 2023-02-09 20:54:36,585 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-top 2023-02-09 20:54:36,589 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-top 2023-02-09 20:54:36,592 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:36,592 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:36,595 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:36,595 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:36,598 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:36,598 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:36,598 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(310): Store=68761db75920bcee7891cf83b0ee94c4/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:36,598 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,599 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2 2023-02-09 20:54:36,599 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2 2023-02-09 20:54:36,599 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68761db75920bcee7891cf83b0ee94c4 columnFamilyName cf2 2023-02-09 20:54:36,608 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-top 2023-02-09 20:54:36,611 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-top 2023-02-09 20:54:36,614 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,615 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-top 2023-02-09 20:54:36,617 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-top 2023-02-09 20:54:36,620 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:36,620 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:36,623 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:36,623 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:36,625 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:36,625 DEBUG [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:36,626 INFO [StoreOpener-68761db75920bcee7891cf83b0ee94c4-1] regionserver.HStore(310): Store=68761db75920bcee7891cf83b0ee94c4/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:36,626 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,627 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:36,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:36,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 68761db75920bcee7891cf83b0ee94c4; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65717895, jitterRate=-0.02072705328464508}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:36,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 68761db75920bcee7891cf83b0ee94c4: Running coprocessor pre-open hook at 1675976076565Writing region info on filesystem at 1675976076565Initializing all the Stores at 1675976076566 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976076566Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976076566Cleaning up temporary data from old regions at 1675976076628 (+62 ms)Cleaning up detritus from prior splits at 1675976076628Running coprocessor post-open hooks at 1675976076630 (+2 ms)Region opened successfully at 1675976076630 2023-02-09 20:54:36,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4., pid=77, masterSystemTime=1675976076561 2023-02-09 20:54:36,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:36,632 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:36,632 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:36,632 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,632 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,632 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1912): 68761db75920bcee7891cf83b0ee94c4/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:36,632 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1912): 68761db75920bcee7891cf83b0ee94c4/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:36,632 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 68761db75920bcee7891cf83b0ee94c4/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,632 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 68761db75920bcee7891cf83b0ee94c4/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,633 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp, totalSize=34.8 K 2023-02-09 20:54:36,633 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp, totalSize=34.8 K 2023-02-09 20:54:36,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,633 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976074086 2023-02-09 20:54:36,633 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976074086 2023-02-09 20:54:36,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,634 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976074276 2023-02-09 20:54:36,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,634 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=68761db75920bcee7891cf83b0ee94c4, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c61416884e2b4074206df3e8b1ddee75, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.', STARTKEY => '', ENDKEY => '60'} 2023-02-09 20:54:36,634 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076634"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976076634"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976076634"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976076634"}]},"ts":"1675976076634"} 2023-02-09 20:54:36,634 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976074478 2023-02-09 20:54:36,634 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976074276 2023-02-09 20:54:36,635 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1675976075204 2023-02-09 20:54:36,635 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976074478 2023-02-09 20:54:36,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:36,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:36,635 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,635 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1675976075204 2023-02-09 20:54:36,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,636 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,636 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,636 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,636 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,637 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=76 2023-02-09 20:54:36,638 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; OpenRegionProcedure 68761db75920bcee7891cf83b0ee94c4, server=jenkins-hbase9.apache.org,36111,1675976048792 in 226 msec 2023-02-09 20:54:36,639 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1 2023-02-09 20:54:36,639 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1 2023-02-09 20:54:36,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, ASSIGN in 384 msec 2023-02-09 20:54:36,640 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c61416884e2b4074206df3e8b1ddee75 columnFamilyName cf1 2023-02-09 20:54:36,651 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,652 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-bottom 2023-02-09 20:54:36,652 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): 68761db75920bcee7891cf83b0ee94c4#cf2#compaction#72 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,653 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): 68761db75920bcee7891cf83b0ee94c4#cf1#compaction#73 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,656 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-bottom 2023-02-09 20:54:36,658 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-bottom 2023-02-09 20:54:36,663 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-bottom 2023-02-09 20:54:36,666 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d 2023-02-09 20:54:36,669 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3 2023-02-09 20:54:36,677 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b 2023-02-09 20:54:36,677 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(310): Store=c61416884e2b4074206df3e8b1ddee75/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:36,677 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,678 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2 2023-02-09 20:54:36,678 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2 2023-02-09 20:54:36,678 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c61416884e2b4074206df3e8b1ddee75 columnFamilyName cf2 2023-02-09 20:54:36,679 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 788bf394a6df470f8a3b33c14cbbd742: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,679 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 134135b829e74c7ca77fb16e863e8d6a: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,679 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp/cf1/788bf394a6df470f8a3b33c14cbbd742 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/788bf394a6df470f8a3b33c14cbbd742 2023-02-09 20:54:36,680 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp/cf2/134135b829e74c7ca77fb16e863e8d6a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/134135b829e74c7ca77fb16e863e8d6a 2023-02-09 20:54:36,685 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 788bf394a6df470f8a3b33c14cbbd742: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,685 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 134135b829e74c7ca77fb16e863e8d6a: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,686 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 68761db75920bcee7891cf83b0ee94c4/cf1 of 68761db75920bcee7891cf83b0ee94c4 into 788bf394a6df470f8a3b33c14cbbd742(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,686 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 68761db75920bcee7891cf83b0ee94c4: 2023-02-09 20:54:36,686 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4., storeName=68761db75920bcee7891cf83b0ee94c4/cf1, priority=9, startTime=1675976076631; duration=0sec 2023-02-09 20:54:36,686 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,687 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 68761db75920bcee7891cf83b0ee94c4/cf2 of 68761db75920bcee7891cf83b0ee94c4 into 134135b829e74c7ca77fb16e863e8d6a(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,687 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 68761db75920bcee7891cf83b0ee94c4: 2023-02-09 20:54:36,687 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4., storeName=68761db75920bcee7891cf83b0ee94c4/cf2, priority=9, startTime=1675976076632; duration=0sec 2023-02-09 20:54:36,687 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,689 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-bottom 2023-02-09 20:54:36,692 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-bottom 2023-02-09 20:54:36,695 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:36,696 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-bottom 2023-02-09 20:54:36,699 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-bottom 2023-02-09 20:54:36,703 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6 2023-02-09 20:54:36,708 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83 2023-02-09 20:54:36,713 DEBUG [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f 2023-02-09 20:54:36,713 INFO [StoreOpener-c61416884e2b4074206df3e8b1ddee75-1] regionserver.HStore(310): Store=c61416884e2b4074206df3e8b1ddee75/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:36,714 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:54:36,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:36,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened c61416884e2b4074206df3e8b1ddee75; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64550855, jitterRate=-0.03811730444431305}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:54:36,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for c61416884e2b4074206df3e8b1ddee75: Running coprocessor pre-open hook at 1675976076636Writing region info on filesystem at 1675976076636Initializing all the Stores at 1675976076637 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976076637Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976076638 (+1 ms)Cleaning up temporary data from old regions at 1675976076715 (+77 ms)Cleaning up detritus from prior splits at 1675976076716 (+1 ms)Running coprocessor post-open hooks at 1675976076718 (+2 ms)Region opened successfully at 1675976076718 2023-02-09 20:54:36,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75., pid=78, masterSystemTime=1675976076561 2023-02-09 20:54:36,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,720 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:36,720 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:36,720 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:54:36,720 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,720 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,721 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1912): c61416884e2b4074206df3e8b1ddee75/cf1 is initiating minor compaction (all files) 2023-02-09 20:54:36,721 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1912): c61416884e2b4074206df3e8b1ddee75/cf2 is initiating minor compaction (all files) 2023-02-09 20:54:36,721 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of c61416884e2b4074206df3e8b1ddee75/cf1 in testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,721 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of c61416884e2b4074206df3e8b1ddee75/cf2 in testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,721 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp, totalSize=35.0 K 2023-02-09 20:54:36,721 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp, totalSize=35.0 K 2023-02-09 20:54:36,722 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976074086 2023-02-09 20:54:36,722 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976074086 2023-02-09 20:54:36,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,723 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976074276 2023-02-09 20:54:36,723 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=c61416884e2b4074206df3e8b1ddee75, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,723 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976074276 2023-02-09 20:54:36,723 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076723"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976076723"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976076723"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976076723"}]},"ts":"1675976076723"} 2023-02-09 20:54:36,723 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976074478 2023-02-09 20:54:36,724 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976074478 2023-02-09 20:54:36,724 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976074664 2023-02-09 20:54:36,724 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976074664 2023-02-09 20:54:36,724 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976074843 2023-02-09 20:54:36,724 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976074843 2023-02-09 20:54:36,725 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976075024 2023-02-09 20:54:36,725 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976075024 2023-02-09 20:54:36,725 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976075204 2023-02-09 20:54:36,726 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976075204 2023-02-09 20:54:36,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=78, resume processing ppid=75 2023-02-09 20:54:36,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=75, state=SUCCESS; OpenRegionProcedure c61416884e2b4074206df3e8b1ddee75, server=jenkins-hbase9.apache.org,36111,1675976048792 in 313 msec 2023-02-09 20:54:36,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=72 2023-02-09 20:54:36,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, ASSIGN in 475 msec 2023-02-09 20:54:36,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 in 971 msec 2023-02-09 20:54:36,746 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): c61416884e2b4074206df3e8b1ddee75#cf2#compaction#74 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,747 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): c61416884e2b4074206df3e8b1ddee75#cf1#compaction#75 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,773 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp/cf1/d20a8267b68a4fcb906d37f448323f45 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/d20a8267b68a4fcb906d37f448323f45 2023-02-09 20:54:36,773 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp/cf2/de658a9f4c2946148b252a06d3da5268 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/de658a9f4c2946148b252a06d3da5268 2023-02-09 20:54:36,777 DEBUG [Listener at localhost/45551] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-09 20:54:36,779 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c61416884e2b4074206df3e8b1ddee75/cf1 of c61416884e2b4074206df3e8b1ddee75 into d20a8267b68a4fcb906d37f448323f45(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,780 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for c61416884e2b4074206df3e8b1ddee75: 2023-02-09 20:54:36,780 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75., storeName=c61416884e2b4074206df3e8b1ddee75/cf1, priority=9, startTime=1675976076719; duration=0sec 2023-02-09 20:54:36,780 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,780 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c61416884e2b4074206df3e8b1ddee75/cf2 of c61416884e2b4074206df3e8b1ddee75 into de658a9f4c2946148b252a06d3da5268(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,780 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for c61416884e2b4074206df3e8b1ddee75: 2023-02-09 20:54:36,780 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75., storeName=c61416884e2b4074206df3e8b1ddee75/cf2, priority=9, startTime=1675976076720; duration=0sec 2023-02-09 20:54:36,780 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,786 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:54:36,786 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37438, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-09 20:54:36,787 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,788 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-09 20:54:36,788 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HStore(1912): c61416884e2b4074206df3e8b1ddee75/cf1 is initiating major compaction (all files) 2023-02-09 20:54:36,788 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@16972427; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,788 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of c61416884e2b4074206df3e8b1ddee75/cf1 in testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,789 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-09 20:54:36,789 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/d20a8267b68a4fcb906d37f448323f45] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp, totalSize=6.8 K 2023-02-09 20:54:36,789 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HStore(1912): c61416884e2b4074206df3e8b1ddee75/cf2 is initiating major compaction (all files) 2023-02-09 20:54:36,789 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@3f07aa45; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:36,789 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting d20a8267b68a4fcb906d37f448323f45, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976074086 2023-02-09 20:54:36,791 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of c61416884e2b4074206df3e8b1ddee75/cf2 in testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:36,791 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/de658a9f4c2946148b252a06d3da5268] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp, totalSize=6.8 K 2023-02-09 20:54:36,791 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting de658a9f4c2946148b252a06d3da5268, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976074086 2023-02-09 20:54:36,794 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): c61416884e2b4074206df3e8b1ddee75#cf1#compaction#76 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,794 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,794 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-09 20:54:36,794 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.HStore(1912): 68761db75920bcee7891cf83b0ee94c4/cf1 is initiating major compaction (all files) 2023-02-09 20:54:36,794 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@1f00efa; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:36,795 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-02-09 20:54:36,795 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.HStore(1912): 68761db75920bcee7891cf83b0ee94c4/cf2 is initiating major compaction (all files) 2023-02-09 20:54:36,795 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4c5e4413; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-09 20:54:36,796 INFO [Listener at localhost/45551] hbase.Waiter(180): Waiting up to [6,000] milli-secs(wait.for.ratio=[1]) 2023-02-09 20:54:36,799 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): c61416884e2b4074206df3e8b1ddee75#cf2#compaction#77 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,822 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. is not mergeable because it has references 2023-02-09 20:54:36,823 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. is not mergeable because it has references 2023-02-09 20:54:36,832 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp/cf1/a96838feb7ce415eae2c980d47108399 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/a96838feb7ce415eae2c980d47108399 2023-02-09 20:54:36,838 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in c61416884e2b4074206df3e8b1ddee75/cf1 of c61416884e2b4074206df3e8b1ddee75 into a96838feb7ce415eae2c980d47108399(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,838 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for c61416884e2b4074206df3e8b1ddee75: 2023-02-09 20:54:36,840 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75., storeName=cf1, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1675976076788; duration=0sec 2023-02-09 20:54:36,840 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-02-09 20:54:36,840 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/.tmp/cf2/9c7664709a284f6b8e926e04a0efd74b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/9c7664709a284f6b8e926e04a0efd74b 2023-02-09 20:54:36,840 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 68761db75920bcee7891cf83b0ee94c4/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,840 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/788bf394a6df470f8a3b33c14cbbd742] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp, totalSize=5.0 K 2023-02-09 20:54:36,840 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] compactions.Compactor(207): Compacting 788bf394a6df470f8a3b33c14cbbd742, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,844 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] throttle.PressureAwareThroughputController(145): 68761db75920bcee7891cf83b0ee94c4#cf1#compaction#78 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,845 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in c61416884e2b4074206df3e8b1ddee75/cf2 of c61416884e2b4074206df3e8b1ddee75 into 9c7664709a284f6b8e926e04a0efd74b(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,845 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for c61416884e2b4074206df3e8b1ddee75: 2023-02-09 20:54:36,845 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75., storeName=cf2, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1675976076789; duration=0sec 2023-02-09 20:54:36,845 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:54:36,845 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 68761db75920bcee7891cf83b0ee94c4/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:36,846 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/134135b829e74c7ca77fb16e863e8d6a] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp, totalSize=5.0 K 2023-02-09 20:54:36,846 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] compactions.Compactor(207): Compacting 134135b829e74c7ca77fb16e863e8d6a, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-02-09 20:54:36,851 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] throttle.PressureAwareThroughputController(145): 68761db75920bcee7891cf83b0ee94c4#cf2#compaction#79 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:54:36,857 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 6aff3e8eea024cc68e9366fa79fda4fd: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,858 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp/cf1/6aff3e8eea024cc68e9366fa79fda4fd as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/6aff3e8eea024cc68e9366fa79fda4fd 2023-02-09 20:54:36,861 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for e4dd356b299347a3b08d93c5a986dc29: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,862 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/.tmp/cf2/e4dd356b299347a3b08d93c5a986dc29 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e4dd356b299347a3b08d93c5a986dc29 2023-02-09 20:54:36,862 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 6aff3e8eea024cc68e9366fa79fda4fd: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,863 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 68761db75920bcee7891cf83b0ee94c4/cf1 of 68761db75920bcee7891cf83b0ee94c4 into 6aff3e8eea024cc68e9366fa79fda4fd(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,863 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 68761db75920bcee7891cf83b0ee94c4: 2023-02-09 20:54:36,863 INFO [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4., storeName=cf1, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1675976076794; duration=0sec 2023-02-09 20:54:36,863 DEBUG [RS:1;jenkins-hbase9:36111-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,866 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for e4dd356b299347a3b08d93c5a986dc29: NONE, but ROW specified in column family configuration 2023-02-09 20:54:36,866 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 68761db75920bcee7891cf83b0ee94c4/cf2 of 68761db75920bcee7891cf83b0ee94c4 into e4dd356b299347a3b08d93c5a986dc29(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:54:36,867 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 68761db75920bcee7891cf83b0ee94c4: 2023-02-09 20:54:36,867 INFO [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4., storeName=cf2, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1675976076795; duration=0sec 2023-02-09 20:54:36,867 DEBUG [RS:1;jenkins-hbase9:36111-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:54:36,930 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. is not mergeable because it has references 2023-02-09 20:54:36,930 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=36111] regionserver.HRegion(1450): Region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. is not mergeable because it has references 2023-02-09 20:54:36,936 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '60', inclusive 2023-02-09 20:54:36,938 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:36,939 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:36,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:36,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=79, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:36,943 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976076943"}]},"ts":"1675976076943"} 2023-02-09 20:54:36,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-09 20:54:36,944 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLING in hbase:meta 2023-02-09 20:54:36,945 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLING 2023-02-09 20:54:36,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, UNASSIGN}, {pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, UNASSIGN}] 2023-02-09 20:54:36,947 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, UNASSIGN 2023-02-09 20:54:36,947 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=68761db75920bcee7891cf83b0ee94c4, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,947 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076947"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976076947"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976076947"}]},"ts":"1675976076947"} 2023-02-09 20:54:36,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=82, ppid=81, state=RUNNABLE; CloseRegionProcedure 68761db75920bcee7891cf83b0ee94c4, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:36,950 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, UNASSIGN 2023-02-09 20:54:36,951 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=c61416884e2b4074206df3e8b1ddee75, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:54:36,951 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976076951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976076951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976076951"}]},"ts":"1675976076951"} 2023-02-09 20:54:36,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure c61416884e2b4074206df3e8b1ddee75, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:54:37,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-09 20:54:37,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:37,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing c61416884e2b4074206df3e8b1ddee75, disabling compactions & flushes 2023-02-09 20:54:37,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:37,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:37,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. after waiting 0 ms 2023-02-09 20:54:37,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:37,101 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/d20a8267b68a4fcb906d37f448323f45, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-bottom] to archive 2023-02-09 20:54:37,101 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:37,103 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,104 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,105 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,106 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-eb99041bef0d40caadb25334afcded5b 2023-02-09 20:54:37,107 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-a37abf412cfe464481967bf740c784c3 2023-02-09 20:54:37,108 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-13b7bfe86c384e1893a6aab3f2eb901d 2023-02-09 20:54:37,109 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/d20a8267b68a4fcb906d37f448323f45 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/d20a8267b68a4fcb906d37f448323f45 2023-02-09 20:54:37,110 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,111 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/de658a9f4c2946148b252a06d3da5268, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-bottom] to archive 2023-02-09 20:54:37,112 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:37,113 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,114 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,115 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,116 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-efe729ad4ed549409aae127a40ad112f 2023-02-09 20:54:37,117 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-49b11d79c2fb431290af079dd778da83 2023-02-09 20:54:37,118 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-170abd2c6fc148199da85887acb915a6 2023-02-09 20:54:37,119 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/de658a9f4c2946148b252a06d3da5268 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/de658a9f4c2946148b252a06d3da5268 2023-02-09 20:54:37,119 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-02-09 20:54:37,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:37,124 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75. 2023-02-09 20:54:37,124 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for c61416884e2b4074206df3e8b1ddee75: Waiting for close lock at 1675976077100Running coprocessor pre-close hooks at 1675976077100Disabling compacts and flushes for region at 1675976077100Disabling writes for close at 1675976077100Writing region close event to WAL at 1675976077120 (+20 ms)Running coprocessor post-close hooks at 1675976077123 (+3 ms)Closed at 1675976077124 (+1 ms) 2023-02-09 20:54:37,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:37,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:37,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 68761db75920bcee7891cf83b0ee94c4, disabling compactions & flushes 2023-02-09 20:54:37,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:37,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:37,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. after waiting 0 ms 2023-02-09 20:54:37,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:37,126 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=c61416884e2b4074206df3e8b1ddee75, regionState=CLOSED 2023-02-09 20:54:37,126 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976077126"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976077126"}]},"ts":"1675976077126"} 2023-02-09 20:54:37,127 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/788bf394a6df470f8a3b33c14cbbd742, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45] to archive 2023-02-09 20:54:37,127 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:37,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-02-09 20:54:37,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure c61416884e2b4074206df3e8b1ddee75, server=jenkins-hbase9.apache.org,36111,1675976048792 in 175 msec 2023-02-09 20:54:37,129 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/55af41f66c7f464ca7ecac7c074da705.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=c61416884e2b4074206df3e8b1ddee75, UNASSIGN in 183 msec 2023-02-09 20:54:37,130 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/39985665b359468fbdd6d457d3bdbdac.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,131 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/ee51309bb4224e1eb5b06abd333fa570.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,132 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/1eda5274095b477093ba8a1a5dc3733c.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,133 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:37,134 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:37,134 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/788bf394a6df470f8a3b33c14cbbd742 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/788bf394a6df470f8a3b33c14cbbd742 2023-02-09 20:54:37,135 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:37,137 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/134135b829e74c7ca77fb16e863e8d6a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea] to archive 2023-02-09 20:54:37,137 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:54:37,138 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/d2d38e5493664d76b20f718170beb5dc.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,139 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/f9e12424936a4c939783032b70326d7e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,140 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/20be9bf7ed3646568a2b2e12754face0.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,141 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e0996e212b8b4f54b155564d3c53b37e.f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,142 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:37,143 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:37,144 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/134135b829e74c7ca77fb16e863e8d6a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/134135b829e74c7ca77fb16e863e8d6a 2023-02-09 20:54:37,145 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/testSplitTableRegionDeletedRowsDaughter=f914af6ac487ff13c2af3eebff56a1b1-3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:37,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-02-09 20:54:37,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:37,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4. 2023-02-09 20:54:37,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 68761db75920bcee7891cf83b0ee94c4: Waiting for close lock at 1675976077126Running coprocessor pre-close hooks at 1675976077126Disabling compacts and flushes for region at 1675976077126Disabling writes for close at 1675976077126Writing region close event to WAL at 1675976077146 (+20 ms)Running coprocessor post-close hooks at 1675976077148 (+2 ms)Closed at 1675976077149 (+1 ms) 2023-02-09 20:54:37,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:37,151 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=68761db75920bcee7891cf83b0ee94c4, regionState=CLOSED 2023-02-09 20:54:37,151 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1675976077151"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976077151"}]},"ts":"1675976077151"} 2023-02-09 20:54:37,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=81 2023-02-09 20:54:37,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=81, state=SUCCESS; CloseRegionProcedure 68761db75920bcee7891cf83b0ee94c4, server=jenkins-hbase9.apache.org,36111,1675976048792 in 204 msec 2023-02-09 20:54:37,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=79 2023-02-09 20:54:37,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=68761db75920bcee7891cf83b0ee94c4, UNASSIGN in 207 msec 2023-02-09 20:54:37,155 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976077155"}]},"ts":"1675976077155"} 2023-02-09 20:54:37,156 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLED in hbase:meta 2023-02-09 20:54:37,157 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLED 2023-02-09 20:54:37,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 219 msec 2023-02-09 20:54:37,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-02-09 20:54:37,245 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 79 completed 2023-02-09 20:54:37,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,248 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,249 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=84, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-02-09 20:54:37,253 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,253 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:37,253 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:37,255 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/recovered.edits] 2023-02-09 20:54:37,255 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/recovered.edits] 2023-02-09 20:54:37,255 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/recovered.edits] 2023-02-09 20:54:37,257 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/015afc3c1d264f8e8f263f6b2f0b6da6 2023-02-09 20:54:37,257 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/6aff3e8eea024cc68e9366fa79fda4fd to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf1/6aff3e8eea024cc68e9366fa79fda4fd 2023-02-09 20:54:37,258 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/a96838feb7ce415eae2c980d47108399 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf1/a96838feb7ce415eae2c980d47108399 2023-02-09 20:54:37,259 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/026137f74cfd4840a1dbbca1c112358b 2023-02-09 20:54:37,260 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e4dd356b299347a3b08d93c5a986dc29 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/cf2/e4dd356b299347a3b08d93c5a986dc29 2023-02-09 20:54:37,260 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/9c7664709a284f6b8e926e04a0efd74b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/cf2/9c7664709a284f6b8e926e04a0efd74b 2023-02-09 20:54:37,260 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/13b7bfe86c384e1893a6aab3f2eb901d 2023-02-09 20:54:37,261 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/1eda5274095b477093ba8a1a5dc3733c 2023-02-09 20:54:37,263 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/39985665b359468fbdd6d457d3bdbdac 2023-02-09 20:54:37,263 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/recovered.edits/113.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4/recovered.edits/113.seqid 2023-02-09 20:54:37,263 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/recovered.edits/113.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75/recovered.edits/113.seqid 2023-02-09 20:54:37,263 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:54:37,264 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/c61416884e2b4074206df3e8b1ddee75 2023-02-09 20:54:37,264 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/55af41f66c7f464ca7ecac7c074da705 2023-02-09 20:54:37,265 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/a37abf412cfe464481967bf740c784c3 2023-02-09 20:54:37,266 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/d922d8d2f7b24ed38169e021ec403a45 2023-02-09 20:54:37,267 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/eb99041bef0d40caadb25334afcded5b 2023-02-09 20:54:37,268 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf1/ee51309bb4224e1eb5b06abd333fa570 2023-02-09 20:54:37,270 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/170abd2c6fc148199da85887acb915a6 2023-02-09 20:54:37,271 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/20be9bf7ed3646568a2b2e12754face0 2023-02-09 20:54:37,272 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/3e4e36afdcc743c49295f8a3fdad16ea 2023-02-09 20:54:37,273 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/49b11d79c2fb431290af079dd778da83 2023-02-09 20:54:37,273 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/5b1a4fb0efa84f7c88035445ebf759ec 2023-02-09 20:54:37,274 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d2d38e5493664d76b20f718170beb5dc 2023-02-09 20:54:37,275 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/d349cf4b521c483d902dd7bd1fb31fea 2023-02-09 20:54:37,276 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/e0996e212b8b4f54b155564d3c53b37e 2023-02-09 20:54:37,277 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/efe729ad4ed549409aae127a40ad112f 2023-02-09 20:54:37,279 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/cf2/f9e12424936a4c939783032b70326d7e 2023-02-09 20:54:37,281 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/recovered.edits/106.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1/recovered.edits/106.seqid 2023-02-09 20:54:37,281 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/f914af6ac487ff13c2af3eebff56a1b1 2023-02-09 20:54:37,282 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-02-09 20:54:37,283 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=84, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,285 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionDeletedRowsDaughter from hbase:meta 2023-02-09 20:54:37,287 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionDeletedRowsDaughter' descriptor. 2023-02-09 20:54:37,288 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=84, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,288 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionDeletedRowsDaughter' from region states. 2023-02-09 20:54:37,288 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976077288"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:37,288 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976077288"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:37,288 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976077288"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:37,289 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:54:37,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f914af6ac487ff13c2af3eebff56a1b1, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1675976073582.f914af6ac487ff13c2af3eebff56a1b1.', STARTKEY => '', ENDKEY => ''}, {ENCODED => c61416884e2b4074206df3e8b1ddee75, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1675976075759.c61416884e2b4074206df3e8b1ddee75.', STARTKEY => '', ENDKEY => '60'}, {ENCODED => 68761db75920bcee7891cf83b0ee94c4, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1675976075759.68761db75920bcee7891cf83b0ee94c4.', STARTKEY => '60', ENDKEY => ''}] 2023-02-09 20:54:37,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionDeletedRowsDaughter' as deleted. 2023-02-09 20:54:37,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976077290"}]},"ts":"9223372036854775807"} 2023-02-09 20:54:37,291 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionDeletedRowsDaughter state from META 2023-02-09 20:54:37,293 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=84, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:54:37,294 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 47 msec 2023-02-09 20:54:37,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-02-09 20:54:37,350 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 84 completed 2023-02-09 20:54:37,366 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=411 (was 407) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:57120 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:50020 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49386 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:50028 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49188 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49898 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:57142 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49142 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:57788 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49180 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49874 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:50084 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49460 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49894 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:50086 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1194637468_15 at /127.0.0.1:49088 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7e2f5f25-metaLookup-shared--pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=852 (was 847) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=198 (was 215), ProcessCount=168 (was 168), AvailableMemoryMB=10738 (was 10795) 2023-02-09 20:54:37,381 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=411, OpenFileDescriptor=852, MaxFileDescriptor=60000, SystemLoadAverage=198, ProcessCount=168, AvailableMemoryMB=10737 2023-02-09 20:54:37,381 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:54:37,385 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:54:37,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:54:37,386 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:54:37,387 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:54:37,388 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:54:37,389 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,390 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a empty. 2023-02-09 20:54:37,390 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,390 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-02-09 20:54:37,401 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:37,402 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(7671): creating {ENCODED => 8b30263dbcee1c353548dc8aaf90953a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1603): Closing 8b30263dbcee1c353548dc8aaf90953a, disabling compactions & flushes 2023-02-09 20:54:37,411 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. after waiting 0 ms 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,411 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,411 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1557): Region close journal for 8b30263dbcee1c353548dc8aaf90953a: Waiting for close lock at 1675976077411Disabling compacts and flushes for region at 1675976077411Disabling writes for close at 1675976077411Writing region close event to WAL at 1675976077411Closed at 1675976077411 2023-02-09 20:54:37,413 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:54:37,414 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976077414"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976077414"}]},"ts":"1675976077414"} 2023-02-09 20:54:37,415 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:54:37,416 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:54:37,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976077416"}]},"ts":"1675976077416"} 2023-02-09 20:54:37,417 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLING in hbase:meta 2023-02-09 20:54:37,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:37,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:37,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:37,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:37,420 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:54:37,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, ASSIGN}] 2023-02-09 20:54:37,422 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, ASSIGN 2023-02-09 20:54:37,423 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,39321,1675976048743; forceNewPlan=false, retain=false 2023-02-09 20:54:37,573 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:37,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:37,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976077575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976077575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976077575"}]},"ts":"1675976077575"} 2023-02-09 20:54:37,577 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=86, state=RUNNABLE; OpenRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:37,734 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8b30263dbcee1c353548dc8aaf90953a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:37,734 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:37,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:37,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,736 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,737 DEBUG [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:54:37,737 DEBUG [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:54:37,737 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8b30263dbcee1c353548dc8aaf90953a columnFamilyName cf1 2023-02-09 20:54:37,738 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] regionserver.HStore(310): Store=8b30263dbcee1c353548dc8aaf90953a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:37,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:37,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:54:37,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 8b30263dbcee1c353548dc8aaf90953a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61587152, jitterRate=-0.08227992057800293}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 20:54:37,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 8b30263dbcee1c353548dc8aaf90953a: Running coprocessor pre-open hook at 1675976077735Writing region info on filesystem at 1675976077735Initializing all the Stores at 1675976077735Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976077735Cleaning up temporary data from old regions at 1675976077739 (+4 ms)Cleaning up detritus from prior splits at 1675976077739Running coprocessor post-open hooks at 1675976077742 (+3 ms)Region opened successfully at 1675976077742 2023-02-09 20:54:37,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a., pid=87, masterSystemTime=1675976077728 2023-02-09 20:54:37,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:37,745 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:37,745 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976077745"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976077745"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976077745"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976077745"}]},"ts":"1675976077745"} 2023-02-09 20:54:37,747 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=86 2023-02-09 20:54:37,747 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=86, state=SUCCESS; OpenRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743 in 169 msec 2023-02-09 20:54:37,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=85 2023-02-09 20:54:37,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=85, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, ASSIGN in 327 msec 2023-02-09 20:54:37,749 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:54:37,749 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976077749"}]},"ts":"1675976077749"} 2023-02-09 20:54:37,750 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLED in hbase:meta 2023-02-09 20:54:37,757 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:54:37,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica in 372 msec 2023-02-09 20:54:37,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$12(2441): Client=jenkins//172.31.2.10 modify table testRollbackForSplitTableRegionWithReplica from 'testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'testRollbackForSplitTableRegionWithReplica', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}}, {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-09 20:54:37,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=88, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:54:37,914 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-02-09 20:54:37,915 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-02-09 20:54:37,915 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-02-09 20:54:37,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica}] 2023-02-09 20:54:37,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, REOPEN/MOVE}] 2023-02-09 20:54:37,930 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, REOPEN/MOVE 2023-02-09 20:54:37,930 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:37,930 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976077930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976077930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976077930"}]},"ts":"1675976077930"} 2023-02-09 20:54:37,931 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=91, ppid=90, state=RUNNABLE; CloseRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:38,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 8b30263dbcee1c353548dc8aaf90953a, disabling compactions & flushes 2023-02-09 20:54:38,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. after waiting 0 ms 2023-02-09 20:54:38,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-09 20:54:38,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:54:38,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 8b30263dbcee1c353548dc8aaf90953a: Waiting for close lock at 1675976078083Running coprocessor pre-close hooks at 1675976078083Disabling compacts and flushes for region at 1675976078083Disabling writes for close at 1675976078084 (+1 ms)Writing region close event to WAL at 1675976078085 (+1 ms)Running coprocessor post-close hooks at 1675976078088 (+3 ms)Closed at 1675976078088 2023-02-09 20:54:38,088 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8b30263dbcee1c353548dc8aaf90953a to self. 2023-02-09 20:54:38,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,090 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=CLOSED 2023-02-09 20:54:38,090 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976078090"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976078090"}]},"ts":"1675976078090"} 2023-02-09 20:54:38,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=91, resume processing ppid=90 2023-02-09 20:54:38,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, ppid=90, state=SUCCESS; CloseRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743 in 160 msec 2023-02-09 20:54:38,093 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase9.apache.org,39321,1675976048743; forceNewPlan=false, retain=true 2023-02-09 20:54:38,243 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:54:38,244 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:38,244 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976078244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976078244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976078244"}]},"ts":"1675976078244"} 2023-02-09 20:54:38,245 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=92, ppid=90, state=RUNNABLE; OpenRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:54:38,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 8b30263dbcee1c353548dc8aaf90953a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:54:38,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:38,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:38,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,401 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,402 DEBUG [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:54:38,402 DEBUG [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:54:38,403 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8b30263dbcee1c353548dc8aaf90953a columnFamilyName cf1 2023-02-09 20:54:38,403 INFO [StoreOpener-8b30263dbcee1c353548dc8aaf90953a-1] regionserver.HStore(310): Store=8b30263dbcee1c353548dc8aaf90953a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:54:38,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:54:38,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 8b30263dbcee1c353548dc8aaf90953a; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64737677, jitterRate=-0.03533343970775604}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 20:54:38,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 8b30263dbcee1c353548dc8aaf90953a: Running coprocessor pre-open hook at 1675976078400Writing region info on filesystem at 1675976078400Initializing all the Stores at 1675976078401 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976078401Cleaning up temporary data from old regions at 1675976078405 (+4 ms)Cleaning up detritus from prior splits at 1675976078406 (+1 ms)Running coprocessor post-open hooks at 1675976078408 (+2 ms)Region opened successfully at 1675976078408 2023-02-09 20:54:38,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a., pid=92, masterSystemTime=1675976078397 2023-02-09 20:54:38,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:54:38,411 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:54:38,411 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976078411"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976078411"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976078411"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976078411"}]},"ts":"1675976078411"} 2023-02-09 20:54:38,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=92, resume processing ppid=90 2023-02-09 20:54:38,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, ppid=90, state=SUCCESS; OpenRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743 in 167 msec 2023-02-09 20:54:38,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=90, resume processing ppid=89 2023-02-09 20:54:38,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, ppid=89, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, REOPEN/MOVE in 484 msec 2023-02-09 20:54:38,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=89, resume processing ppid=88 2023-02-09 20:54:38,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, ppid=88, state=SUCCESS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica in 499 msec 2023-02-09 20:54:38,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, ASSIGN}] 2023-02-09 20:54:38,421 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, ASSIGN 2023-02-09 20:54:38,422 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-09 20:54:38,572 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:54:38,573 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:54:38,573 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:54:38,573 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:54:38,573 DEBUG [jenkins-hbase9:42381] balancer.BaseLoadBalancer$Cluster(378): Number of tables=3, number of hosts=1, number of racks=1 2023-02-09 20:54:38,574 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=50cb6759e917f82973244c041cb8ef94, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:54:38,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976078574"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1675976078574"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1675976078574"}]},"ts":"1675976078574"} 2023-02-09 20:54:38,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=94, ppid=93, state=RUNNABLE; OpenRegionProcedure 50cb6759e917f82973244c041cb8ef94, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:54:38,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:54:38,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 50cb6759e917f82973244c041cb8ef94, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2023-02-09 20:54:38,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:54:38,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:54:38,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:54:38,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:54:38,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:54:38,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:38,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:38,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:39,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,251 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackForSplitTableRegionWithReplica' 2023-02-09 20:54:40,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:40,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:41,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,066 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:54:42,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,240 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,641 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:42,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,041 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,242 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,643 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:43,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,244 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,645 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:44,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,647 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:45,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:46,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:47,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:48,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:49,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,055 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,356 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,456 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,857 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:50,957 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,057 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,859 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:51,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,360 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,761 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,861 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:52,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,062 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,262 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,362 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,462 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,663 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,763 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:53,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,064 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,364 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:54,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,667 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:55,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,068 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,669 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:56,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,070 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,471 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:57,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:58,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:54:59,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,376 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,877 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:00,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:01,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,380 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:02,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,883 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:03,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:04,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:05,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:06,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:07,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,492 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:08,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:09,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,239 DEBUG [regionserver/jenkins-hbase9:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-09 20:55:10,292 DEBUG [regionserver/jenkins-hbase9:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-09 20:55:10,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,332 DEBUG [regionserver/jenkins-hbase9:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-02-09 20:55:10,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:10,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,398 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,498 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,899 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:11,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:12,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,502 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:13,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,003 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,805 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:14,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:15,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:16,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,510 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:17,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:18,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,514 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:19,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,716 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:20,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:21,919 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:22,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:23,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:24,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:25,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:26,928 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,730 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:27,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:28,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,033 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,734 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:29,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,035 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:30,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,037 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,538 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:31,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,039 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,139 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,239 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:32,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,041 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:33,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,644 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:34,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,545 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,646 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:35,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,547 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:36,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,048 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,349 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,549 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:37,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-09 20:55:38,050 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-09 20:55:38,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-09 20:55:38,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-09 20:55:38,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,852 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:38,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-02-09 20:55:39,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:39,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:40,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:40,109 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:40,110 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:40,111 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:40,111 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:40,111 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:40,114 ERROR [PEWorker-1] assignment.SplitTableRegionProcedure(347): Splitting 8b30263dbcee1c353548dc8aaf90953a, pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up at org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createUnassignProcedures(SplitTableRegionProcedure.java:881) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:298) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:94) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1646) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:40,117 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 exec-time=7 msec 2023-02-09 20:55:40,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:40,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:40,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-02-09 20:55:40,455 INFO [StoreOpener-50cb6759e917f82973244c041cb8ef94-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:55:40,458 DEBUG [StoreOpener-50cb6759e917f82973244c041cb8ef94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:55:40,458 DEBUG [StoreOpener-50cb6759e917f82973244c041cb8ef94-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1 2023-02-09 20:55:40,458 INFO [StoreOpener-50cb6759e917f82973244c041cb8ef94-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 50cb6759e917f82973244c041cb8ef94 columnFamilyName cf1 2023-02-09 20:55:40,459 INFO [StoreOpener-50cb6759e917f82973244c041cb8ef94-1] regionserver.HStore(310): Store=50cb6759e917f82973244c041cb8ef94/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:40,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 50cb6759e917f82973244c041cb8ef94; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61716238, jitterRate=-0.08035638928413391}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-09 20:55:40,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 50cb6759e917f82973244c041cb8ef94: Running coprocessor pre-open hook at 1675976078733Initializing all the Stores at 1675976140455 (+61722 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976140455Running coprocessor post-open hooks at 1675976140460 (+5 ms)Region opened successfully at 1675976140460 2023-02-09 20:55:40,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94., pid=94, masterSystemTime=1675976078730 2023-02-09 20:55:40,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:40,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:40,463 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=50cb6759e917f82973244c041cb8ef94, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:40,463 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976140463"},{"qualifier":"server_0001","vlen":31,"tag":[],"timestamp":"1675976140463"},{"qualifier":"serverstartcode_0001","vlen":8,"tag":[],"timestamp":"1675976140463"},{"qualifier":"seqnumDuringOpen_0001","vlen":8,"tag":[],"timestamp":"1675976140463"}]},"ts":"1675976140463"} 2023-02-09 20:55:40,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=94, resume processing ppid=93 2023-02-09 20:55:40,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=93, state=SUCCESS; OpenRegionProcedure 50cb6759e917f82973244c041cb8ef94, server=jenkins-hbase9.apache.org,43365,1675976048828 in 1 mins, 1.889 sec 2023-02-09 20:55:40,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=88 2023-02-09 20:55:40,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, ASSIGN in 1 mins, 2.046 sec 2023-02-09 20:55:40,469 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica in 1 mins, 2.57 sec 2023-02-09 20:55:40,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] ipc.RpcServer(528): (responseTooSlow): {"call":"ModifyTable(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest)","starttimems":"1675976077893","responsesize":"2","method":"ModifyTable","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest","processingtimems":62576,"client":"172.31.2.10:55528","queuetimems":0,"class":"HMaster"} 2023-02-09 20:55:40,470 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcDuplexHandler(156): Unknown callId: 601, skipping over this response of 3 bytes 2023-02-09 20:55:42,361 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(335): procId=95 exception: org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up 2023-02-09 20:55:42,363 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:42,365 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=96, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-09 20:55:42,376 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976142376"}]},"ts":"1675976142376"} 2023-02-09 20:55:42,377 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLING in hbase:meta 2023-02-09 20:55:42,386 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackForSplitTableRegionWithReplica to state=DISABLING 2023-02-09 20:55:42,387 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, UNASSIGN}, {pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, UNASSIGN}] 2023-02-09 20:55:42,388 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, UNASSIGN 2023-02-09 20:55:42,389 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=50cb6759e917f82973244c041cb8ef94, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:42,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976142389"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1675976142389"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1675976142389"}]},"ts":"1675976142389"} 2023-02-09 20:55:42,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; CloseRegionProcedure 50cb6759e917f82973244c041cb8ef94, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:42,396 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, UNASSIGN 2023-02-09 20:55:42,397 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:42,397 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976142397"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976142397"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976142397"}]},"ts":"1675976142397"} 2023-02-09 20:55:42,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; CloseRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743}] 2023-02-09 20:55:42,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-09 20:55:42,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:55:42,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 50cb6759e917f82973244c041cb8ef94, disabling compactions & flushes 2023-02-09 20:55:42,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:42,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:42,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. after waiting 0 ms 2023-02-09 20:55:42,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:42,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:42,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94. 2023-02-09 20:55:42,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 50cb6759e917f82973244c041cb8ef94: Waiting for close lock at 1675976142544Running coprocessor pre-close hooks at 1675976142544Disabling compacts and flushes for region at 1675976142544Disabling writes for close at 1675976142545 (+1 ms)Writing region close event to WAL at 1675976142546 (+1 ms)Running coprocessor post-close hooks at 1675976142546Closed at 1675976142547 (+1 ms) 2023-02-09 20:55:42,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 50cb6759e917f82973244c041cb8ef94 2023-02-09 20:55:42,549 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=50cb6759e917f82973244c041cb8ef94, regionState=CLOSED 2023-02-09 20:55:42,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:55:42,550 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976142549"},{"qualifier":"state_0001","vlen":6,"tag":[],"timestamp":"1675976142549"}]},"ts":"1675976142549"} 2023-02-09 20:55:42,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 8b30263dbcee1c353548dc8aaf90953a, disabling compactions & flushes 2023-02-09 20:55:42,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:55:42,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:55:42,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. after waiting 0 ms 2023-02-09 20:55:42,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:55:42,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-02-09 20:55:42,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; CloseRegionProcedure 50cb6759e917f82973244c041cb8ef94, server=jenkins-hbase9.apache.org,43365,1675976048828 in 162 msec 2023-02-09 20:55:42,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-02-09 20:55:42,561 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94, UNASSIGN in 171 msec 2023-02-09 20:55:42,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:42,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a. 2023-02-09 20:55:42,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 8b30263dbcee1c353548dc8aaf90953a: Waiting for close lock at 1675976142550Running coprocessor pre-close hooks at 1675976142550Disabling compacts and flushes for region at 1675976142550Disabling writes for close at 1675976142551 (+1 ms)Writing region close event to WAL at 1675976142552 (+1 ms)Running coprocessor post-close hooks at 1675976142561 (+9 ms)Closed at 1675976142562 (+1 ms) 2023-02-09 20:55:42,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:55:42,568 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=8b30263dbcee1c353548dc8aaf90953a, regionState=CLOSED 2023-02-09 20:55:42,568 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1675976142568"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976142568"}]},"ts":"1675976142568"} 2023-02-09 20:55:42,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-02-09 20:55:42,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; CloseRegionProcedure 8b30263dbcee1c353548dc8aaf90953a, server=jenkins-hbase9.apache.org,39321,1675976048743 in 171 msec 2023-02-09 20:55:42,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=96 2023-02-09 20:55:42,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=8b30263dbcee1c353548dc8aaf90953a, UNASSIGN in 185 msec 2023-02-09 20:55:42,582 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976142582"}]},"ts":"1675976142582"} 2023-02-09 20:55:42,583 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLED in hbase:meta 2023-02-09 20:55:42,585 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackForSplitTableRegionWithReplica to state=DISABLED 2023-02-09 20:55:42,587 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica in 221 msec 2023-02-09 20:55:42,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-02-09 20:55:42,673 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 96 completed 2023-02-09 20:55:42,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,676 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,677 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=101, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-02-09 20:55:42,681 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:55:42,683 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits] 2023-02-09 20:55:42,688 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits/7.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a/recovered.edits/7.seqid 2023-02-09 20:55:42,688 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/8b30263dbcee1c353548dc8aaf90953a 2023-02-09 20:55:42,688 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-02-09 20:55:42,690 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=101, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,692 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackForSplitTableRegionWithReplica from hbase:meta 2023-02-09 20:55:42,694 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackForSplitTableRegionWithReplica' descriptor. 2023-02-09 20:55:42,695 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=101, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,695 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackForSplitTableRegionWithReplica' from region states. 2023-02-09 20:55:42,695 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976142695"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:42,695 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976142695"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:42,697 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 2 regions from META 2023-02-09 20:55:42,697 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8b30263dbcee1c353548dc8aaf90953a, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385.8b30263dbcee1c353548dc8aaf90953a.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 50cb6759e917f82973244c041cb8ef94, NAME => 'testRollbackForSplitTableRegionWithReplica,,1675976077385_0001.50cb6759e917f82973244c041cb8ef94.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2023-02-09 20:55:42,697 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackForSplitTableRegionWithReplica' as deleted. 2023-02-09 20:55:42,697 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976142697"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:42,698 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackForSplitTableRegionWithReplica state from META 2023-02-09 20:55:42,700 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=101, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:42,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica in 26 msec 2023-02-09 20:55:42,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-02-09 20:55:42,778 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 101 completed 2023-02-09 20:55:42,790 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=346 (was 411), OpenFileDescriptor=656 (was 852), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=108 (was 198), ProcessCount=168 (was 168), AvailableMemoryMB=10531 (was 10737) 2023-02-09 20:55:42,802 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=347, OpenFileDescriptor=656, MaxFileDescriptor=60000, SystemLoadAverage=108, ProcessCount=168, AvailableMemoryMB=10530 2023-02-09 20:55:42,802 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:42,806 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:55:42,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:55:42,807 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:42,808 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:55:42,809 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:55:42,810 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:42,811 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d empty. 2023-02-09 20:55:42,811 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:42,812 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-02-09 20:55:42,821 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/.tabledesc/.tableinfo.0000000001 2023-02-09 20:55:42,822 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(7671): creating {ENCODED => 3fa2a5b7a7c48573ec7c6ac140bce06d, NAME => 'testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testInvalidSplitKey', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(865): Instantiated testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1603): Closing 3fa2a5b7a7c48573ec7c6ac140bce06d, disabling compactions & flushes 2023-02-09 20:55:42,828 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1625): Closing region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1713): Acquired close lock on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. after waiting 0 ms 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1723): Updates disabled for region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:42,828 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1837): Closed testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:42,828 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1557): Region close journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: Waiting for close lock at 1675976142828Disabling compacts and flushes for region at 1675976142828Disabling writes for close at 1675976142828Writing region close event to WAL at 1675976142828Closed at 1675976142828 2023-02-09 20:55:42,831 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:55:42,832 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1675976142831"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976142831"}]},"ts":"1675976142831"} 2023-02-09 20:55:42,833 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:55:42,834 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:55:42,834 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976142834"}]},"ts":"1675976142834"} 2023-02-09 20:55:42,835 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLING in hbase:meta 2023-02-09 20:55:42,838 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:55:42,839 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:55:42,839 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:55:42,839 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:55:42,839 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:55:42,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, ASSIGN}] 2023-02-09 20:55:42,841 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, ASSIGN 2023-02-09 20:55:42,841 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,36111,1675976048792; forceNewPlan=false, retain=false 2023-02-09 20:55:42,992 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:55:42,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=3fa2a5b7a7c48573ec7c6ac140bce06d, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:42,993 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1675976142993"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976142993"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976142993"}]},"ts":"1675976142993"} 2023-02-09 20:55:42,994 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE; OpenRegionProcedure 3fa2a5b7a7c48573ec7c6ac140bce06d, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:55:43,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3fa2a5b7a7c48573ec7c6ac140bce06d, NAME => 'testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:55:43,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testInvalidSplitKey 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,152 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,154 DEBUG [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1 2023-02-09 20:55:43,154 DEBUG [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1 2023-02-09 20:55:43,154 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fa2a5b7a7c48573ec7c6ac140bce06d columnFamilyName cf1 2023-02-09 20:55:43,155 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] regionserver.HStore(310): Store=3fa2a5b7a7c48573ec7c6ac140bce06d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:43,155 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,156 DEBUG [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2 2023-02-09 20:55:43,156 DEBUG [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2 2023-02-09 20:55:43,157 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fa2a5b7a7c48573ec7c6ac140bce06d columnFamilyName cf2 2023-02-09 20:55:43,157 INFO [StoreOpener-3fa2a5b7a7c48573ec7c6ac140bce06d-1] regionserver.HStore(310): Store=3fa2a5b7a7c48573ec7c6ac140bce06d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:43,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testInvalidSplitKey descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:43,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:43,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:55:43,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 3fa2a5b7a7c48573ec7c6ac140bce06d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60335318, jitterRate=-0.10093370079994202}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:43,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: Running coprocessor pre-open hook at 1675976143152Writing region info on filesystem at 1675976143152Initializing all the Stores at 1675976143152Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976143152Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976143152Cleaning up temporary data from old regions at 1675976143159 (+7 ms)Cleaning up detritus from prior splits at 1675976143160 (+1 ms)Running coprocessor post-open hooks at 1675976143165 (+5 ms)Region opened successfully at 1675976143165 2023-02-09 20:55:43,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d., pid=104, masterSystemTime=1675976143146 2023-02-09 20:55:43,168 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=3fa2a5b7a7c48573ec7c6ac140bce06d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,168 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1675976143168"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976143168"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976143168"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976143168"}]},"ts":"1675976143168"} 2023-02-09 20:55:43,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=103 2023-02-09 20:55:43,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=103, state=SUCCESS; OpenRegionProcedure 3fa2a5b7a7c48573ec7c6ac140bce06d, server=jenkins-hbase9.apache.org,36111,1675976048792 in 175 msec 2023-02-09 20:55:43,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=102 2023-02-09 20:55:43,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=102, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, ASSIGN in 333 msec 2023-02-09 20:55:43,176 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:55:43,176 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976143176"}]},"ts":"1675976143176"} 2023-02-09 20:55:43,178 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLED in hbase:meta 2023-02-09 20:55:43,180 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:55:43,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey in 374 msec 2023-02-09 20:55:43,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:43,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:43,316 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:43,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:43,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:43,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,322 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,322 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,322 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:43,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:43,324 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:43,325 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,325 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:43,327 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,327 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,337 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/3618487b3b644c00afd3c2bb0c6c452b 2023-02-09 20:55:43,348 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/56a8347b164f4515a886306dbd890987 2023-02-09 20:55:43,352 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/3618487b3b644c00afd3c2bb0c6c452b as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/3618487b3b644c00afd3c2bb0c6c452b 2023-02-09 20:55:43,355 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/3618487b3b644c00afd3c2bb0c6c452b, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:43,356 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/56a8347b164f4515a886306dbd890987 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/56a8347b164f4515a886306dbd890987 2023-02-09 20:55:43,359 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/56a8347b164f4515a886306dbd890987, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:43,360 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 33ms, sequenceid=6, compaction requested=false 2023-02-09 20:55:43,360 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testInvalidSplitKey' 2023-02-09 20:55:43,361 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:43,361 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:43,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:43,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,361 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,362 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,362 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,363 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,363 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:43,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662fb53e[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:43,364 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:43,364 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,366 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,366 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:43,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,366 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:43,368 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,368 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,368 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,369 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,369 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:43,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:43,372 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:43,372 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:43,372 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:43,372 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:43,372 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,374 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,375 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,376 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,377 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,377 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,380 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:43,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,379 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,381 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,381 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,480 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:43,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:43,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:43,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:43,493 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:43,494 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,494 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:43,494 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:43,494 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,494 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,496 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,496 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,496 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,496 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,496 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,497 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:43,498 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:43,498 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,498 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:43,499 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,503 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,508 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/b96d19c190414908bca2c419cbd9879d 2023-02-09 20:55:43,518 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/787d0b6bf4144a25add6245a7f174a68 2023-02-09 20:55:43,522 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/b96d19c190414908bca2c419cbd9879d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d 2023-02-09 20:55:43,525 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:43,526 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/787d0b6bf4144a25add6245a7f174a68 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68 2023-02-09 20:55:43,530 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:43,531 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 32ms, sequenceid=19, compaction requested=false 2023-02-09 20:55:43,531 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:43,531 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,531 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:43,532 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:43,532 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,532 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,532 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,533 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,533 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,534 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:43,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138628df[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:43,536 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:43,536 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,539 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,539 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:43,540 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,540 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,540 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,541 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:43,544 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:43,544 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:43,544 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:43,544 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:43,544 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,546 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,547 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,547 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,547 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,549 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,550 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,550 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,554 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,554 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:43,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:43,555 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:43,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,656 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:43,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:43,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:43,670 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:43,670 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,670 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:43,670 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:43,671 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,671 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,672 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,672 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,672 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,672 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:43,674 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:43,674 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,674 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:43,675 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:43,675 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,676 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,676 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,676 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,677 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,677 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,683 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/2d4acdcb21b949fdbad12370991431c5 2023-02-09 20:55:43,694 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea 2023-02-09 20:55:43,698 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/2d4acdcb21b949fdbad12370991431c5 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/2d4acdcb21b949fdbad12370991431c5 2023-02-09 20:55:43,701 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/2d4acdcb21b949fdbad12370991431c5, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:43,702 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea 2023-02-09 20:55:43,705 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:43,706 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 31ms, sequenceid=32, compaction requested=true 2023-02-09 20:55:43,706 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:43,706 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,706 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:43,706 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:43,706 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,706 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,706 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,708 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,708 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,709 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,709 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484d7878[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:43,710 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:43,710 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,712 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,712 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:43,712 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,714 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:43,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:43,717 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:43,717 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:43,717 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:43,717 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:43,717 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,719 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,719 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,720 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,720 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,721 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,722 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,722 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:43,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:43,728 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,728 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:43,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,729 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,829 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:43,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:43,843 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:43,843 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,843 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:43,843 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:43,844 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,844 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,845 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,846 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,846 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:43,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:43,848 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:43,848 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:43,848 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,848 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,848 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:43,847 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,848 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:43,849 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,849 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,850 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,850 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,850 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,850 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,857 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/6b498bbbeada40ad8290e4f76c776ce3 2023-02-09 20:55:43,867 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/e5bbf753c37c4b2d9698d341f38e1a9d 2023-02-09 20:55:43,870 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/6b498bbbeada40ad8290e4f76c776ce3 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/6b498bbbeada40ad8290e4f76c776ce3 2023-02-09 20:55:43,873 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/6b498bbbeada40ad8290e4f76c776ce3, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:43,874 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/e5bbf753c37c4b2d9698d341f38e1a9d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/e5bbf753c37c4b2d9698d341f38e1a9d 2023-02-09 20:55:43,877 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/e5bbf753c37c4b2d9698d341f38e1a9d, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:43,878 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 30ms, sequenceid=45, compaction requested=true 2023-02-09 20:55:43,878 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:43,878 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:43,878 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:43,878 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:43,878 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,878 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:43,878 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:43,881 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,881 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,882 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,882 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad1d353[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:43,884 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:43,884 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,885 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,885 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,885 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,885 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,886 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:43,885 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:43,885 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:43,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:43,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:43,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:43,887 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,887 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,887 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,888 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,888 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:43,891 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:43,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:43,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:43,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:43,891 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:43,893 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:43,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:43,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:43,895 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,895 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:43,897 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,897 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,897 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:43,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:43,900 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:43,900 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:43,901 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,001 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:44,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:44,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:44,014 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:44,014 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,014 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:44,014 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:44,014 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,014 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,016 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,016 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,016 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,017 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,017 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,017 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,018 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,018 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,018 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:44,018 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:44,019 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,028 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/789b390570904f5eb77e4f07f2a50086 2023-02-09 20:55:44,038 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/ee0732b14bd44a7b87cc7745cf61015d 2023-02-09 20:55:44,042 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/789b390570904f5eb77e4f07f2a50086 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/789b390570904f5eb77e4f07f2a50086 2023-02-09 20:55:44,045 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/789b390570904f5eb77e4f07f2a50086, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:44,046 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/ee0732b14bd44a7b87cc7745cf61015d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/ee0732b14bd44a7b87cc7745cf61015d 2023-02-09 20:55:44,049 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/ee0732b14bd44a7b87cc7745cf61015d, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:44,050 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 32ms, sequenceid=58, compaction requested=true 2023-02-09 20:55:44,050 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:44,050 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,050 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:44,050 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:44,050 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,050 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,050 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,052 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,052 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,052 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,052 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@714c1d72[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:44,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:44,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:44,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,056 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,056 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,056 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:44,060 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,060 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,060 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,064 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,064 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,064 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,065 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,065 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,065 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,065 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:44,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:44,068 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:44,068 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:44,068 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:44,068 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:44,068 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,070 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,071 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,072 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,072 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,073 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,076 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,076 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,077 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,177 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:44,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:44,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:44,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:44,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-02-09 20:55:44,189 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-02-09 20:55:44,189 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,189 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-02-09 20:55:44,189 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-02-09 20:55:44,189 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,190 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,191 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,191 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,191 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,191 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-02-09 20:55:44,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:44,193 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:44,193 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. started... 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,193 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,193 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:44,194 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,195 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,195 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,195 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,196 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,196 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,201 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/b2d1eabc96654edfb7d7ce96cb6705bd 2023-02-09 20:55:44,211 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/06a0209329874b32942c00c78c8e8b6c 2023-02-09 20:55:44,214 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/b2d1eabc96654edfb7d7ce96cb6705bd as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b2d1eabc96654edfb7d7ce96cb6705bd 2023-02-09 20:55:44,218 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b2d1eabc96654edfb7d7ce96cb6705bd, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:44,218 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/06a0209329874b32942c00c78c8e8b6c as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/06a0209329874b32942c00c78c8e8b6c 2023-02-09 20:55:44,222 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/06a0209329874b32942c00c78c8e8b6c, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:44,222 INFO [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 29ms, sequenceid=71, compaction requested=true 2023-02-09 20:55:44,222 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:44,222 DEBUG [rs(jenkins-hbase9.apache.org,36111,1675976048792)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,222 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:44,222 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:44,223 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,223 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-02-09 20:55:44,223 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-02-09 20:55:44,225 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,225 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,226 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,226 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-02-09 20:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532e0354[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:44,227 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-02-09 20:55:44,227 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:44,229 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:44,229 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,231 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,231 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-02-09 20:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase9.apache.org,36111,1675976048792': 2023-02-09 20:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-02-09 20:55:44,234 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-02-09 20:55:44,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:44,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:44,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-02-09 20:55:44,234 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:44,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,237 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:44,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,238 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:44,239 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-02-09 20:55:44,240 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,240 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,240 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-02-09 20:55:44,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:44,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:44,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:44,243 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-02-09 20:55:44,244 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-02-09 20:55:44,344 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:44,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:44,352 INFO [Listener at localhost/45551] assignment.SplitTableRegionProcedure(206): testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. splitKey isn't explicitly specified, will try to find a best split key from RS jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.StoreUtils(129): cannot split hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d because midkey is the same as first or last row 2023-02-09 20:55:44,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.StoreUtils(129): cannot split hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68 because midkey is the same as first or last row 2023-02-09 20:55:44,354 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegion(2744): Flushing 3fa2a5b7a7c48573ec7c6ac140bce06d 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:55:44,362 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/e8c636bbba2e45fe8e120a2de0682238 2023-02-09 20:55:44,372 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/de32532d246c4fd580d49b33982e5a78 2023-02-09 20:55:44,376 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf1/e8c636bbba2e45fe8e120a2de0682238 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/e8c636bbba2e45fe8e120a2de0682238 2023-02-09 20:55:44,379 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/e8c636bbba2e45fe8e120a2de0682238, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:44,380 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/.tmp/cf2/de32532d246c4fd580d49b33982e5a78 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/de32532d246c4fd580d49b33982e5a78 2023-02-09 20:55:44,383 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/de32532d246c4fd580d49b33982e5a78, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:44,384 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 3fa2a5b7a7c48573ec7c6ac140bce06d in 30ms, sequenceid=82, compaction requested=true 2023-02-09 20:55:44,384 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.HRegion(2445): Flush status journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: 2023-02-09 20:55:44,385 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.StoreUtils(129): cannot split hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d because midkey is the same as first or last row 2023-02-09 20:55:44,385 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36111] regionserver.StoreUtils(129): cannot split hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68 because midkey is the same as first or last row 2023-02-09 20:55:44,385 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,385 DEBUG [Listener at localhost/45551] assignment.TestSplitTableRegionProcedure(438): Expected Split procedure construction failure: Region not splittable because bestSplitPoint = null, maybe table is too small for auto split. For force split, try specifying split row 2023-02-09 20:55:44,385 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:44,387 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testInvalidSplitKey 2023-02-09 20:55:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testInvalidSplitKey 2023-02-09 20:55:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-09 20:55:44,390 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976144390"}]},"ts":"1675976144390"} 2023-02-09 20:55:44,391 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLING in hbase:meta 2023-02-09 20:55:44,393 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testInvalidSplitKey to state=DISABLING 2023-02-09 20:55:44,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, UNASSIGN}] 2023-02-09 20:55:44,394 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, UNASSIGN 2023-02-09 20:55:44,395 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=3fa2a5b7a7c48573ec7c6ac140bce06d, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:44,395 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1675976144395"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976144395"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976144395"}]},"ts":"1675976144395"} 2023-02-09 20:55:44,396 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=107, ppid=106, state=RUNNABLE; CloseRegionProcedure 3fa2a5b7a7c48573ec7c6ac140bce06d, server=jenkins-hbase9.apache.org,36111,1675976048792}] 2023-02-09 20:55:44,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-09 20:55:44,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:44,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 3fa2a5b7a7c48573ec7c6ac140bce06d, disabling compactions & flushes 2023-02-09 20:55:44,549 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. after waiting 0 ms 2023-02-09 20:55:44,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:55:44,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:44,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d. 2023-02-09 20:55:44,553 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 3fa2a5b7a7c48573ec7c6ac140bce06d: Waiting for close lock at 1675976144549Running coprocessor pre-close hooks at 1675976144549Disabling compacts and flushes for region at 1675976144549Disabling writes for close at 1675976144549Writing region close event to WAL at 1675976144550 (+1 ms)Running coprocessor post-close hooks at 1675976144553 (+3 ms)Closed at 1675976144553 2023-02-09 20:55:44,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:44,555 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=3fa2a5b7a7c48573ec7c6ac140bce06d, regionState=CLOSED 2023-02-09 20:55:44,555 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1675976144555"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976144555"}]},"ts":"1675976144555"} 2023-02-09 20:55:44,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=106 2023-02-09 20:55:44,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=106, state=SUCCESS; CloseRegionProcedure 3fa2a5b7a7c48573ec7c6ac140bce06d, server=jenkins-hbase9.apache.org,36111,1675976048792 in 160 msec 2023-02-09 20:55:44,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-09 20:55:44,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=3fa2a5b7a7c48573ec7c6ac140bce06d, UNASSIGN in 164 msec 2023-02-09 20:55:44,559 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976144559"}]},"ts":"1675976144559"} 2023-02-09 20:55:44,560 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLED in hbase:meta 2023-02-09 20:55:44,562 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testInvalidSplitKey to state=DISABLED 2023-02-09 20:55:44,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey in 176 msec 2023-02-09 20:55:44,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-09 20:55:44,692 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testInvalidSplitKey, procId: 105 completed 2023-02-09 20:55:44,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testInvalidSplitKey 2023-02-09 20:55:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,695 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,696 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=108, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-02-09 20:55:44,698 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:44,700 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/recovered.edits] 2023-02-09 20:55:44,703 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/2d4acdcb21b949fdbad12370991431c5 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/2d4acdcb21b949fdbad12370991431c5 2023-02-09 20:55:44,704 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/3618487b3b644c00afd3c2bb0c6c452b to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/3618487b3b644c00afd3c2bb0c6c452b 2023-02-09 20:55:44,705 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/6b498bbbeada40ad8290e4f76c776ce3 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/6b498bbbeada40ad8290e4f76c776ce3 2023-02-09 20:55:44,706 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/789b390570904f5eb77e4f07f2a50086 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/789b390570904f5eb77e4f07f2a50086 2023-02-09 20:55:44,707 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b2d1eabc96654edfb7d7ce96cb6705bd to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b2d1eabc96654edfb7d7ce96cb6705bd 2023-02-09 20:55:44,708 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/b96d19c190414908bca2c419cbd9879d 2023-02-09 20:55:44,708 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/e8c636bbba2e45fe8e120a2de0682238 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf1/e8c636bbba2e45fe8e120a2de0682238 2023-02-09 20:55:44,710 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/06a0209329874b32942c00c78c8e8b6c to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/06a0209329874b32942c00c78c8e8b6c 2023-02-09 20:55:44,711 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/56a8347b164f4515a886306dbd890987 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/56a8347b164f4515a886306dbd890987 2023-02-09 20:55:44,712 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/787d0b6bf4144a25add6245a7f174a68 2023-02-09 20:55:44,713 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/9dcd7ad2fecd4d7f8e7be9da75ceebea 2023-02-09 20:55:44,714 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/de32532d246c4fd580d49b33982e5a78 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/de32532d246c4fd580d49b33982e5a78 2023-02-09 20:55:44,715 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/e5bbf753c37c4b2d9698d341f38e1a9d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/e5bbf753c37c4b2d9698d341f38e1a9d 2023-02-09 20:55:44,716 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/ee0732b14bd44a7b87cc7745cf61015d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/cf2/ee0732b14bd44a7b87cc7745cf61015d 2023-02-09 20:55:44,718 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d/recovered.edits/85.seqid 2023-02-09 20:55:44,718 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testInvalidSplitKey/3fa2a5b7a7c48573ec7c6ac140bce06d 2023-02-09 20:55:44,718 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-02-09 20:55:44,720 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=108, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,721 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testInvalidSplitKey from hbase:meta 2023-02-09 20:55:44,725 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testInvalidSplitKey' descriptor. 2023-02-09 20:55:44,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=108, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,726 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testInvalidSplitKey' from region states. 2023-02-09 20:55:44,726 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976144726"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:44,727 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-02-09 20:55:44,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3fa2a5b7a7c48573ec7c6ac140bce06d, NAME => 'testInvalidSplitKey,,1675976142806.3fa2a5b7a7c48573ec7c6ac140bce06d.', STARTKEY => '', ENDKEY => ''}] 2023-02-09 20:55:44,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testInvalidSplitKey' as deleted. 2023-02-09 20:55:44,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976144727"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:44,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testInvalidSplitKey state from META 2023-02-09 20:55:44,730 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=108, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:44,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey in 37 msec 2023-02-09 20:55:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-02-09 20:55:44,797 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testInvalidSplitKey, procId: 108 completed 2023-02-09 20:55:44,811 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=368 (was 347) - Thread LEAK? -, OpenFileDescriptor=740 (was 656) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=140 (was 108) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=10526 (was 10530) 2023-02-09 20:55:44,828 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=368, OpenFileDescriptor=740, MaxFileDescriptor=60000, SystemLoadAverage=140, ProcessCount=168, AvailableMemoryMB=10526 2023-02-09 20:55:44,828 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:44,832 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:55:44,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:55:44,834 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:44,836 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:55:44,836 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:55:44,838 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:44,838 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 empty. 2023-02-09 20:55:44,839 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:44,839 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-02-09 20:55:44,852 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-02-09 20:55:44,853 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(7671): creating {ENCODED => ae5b7764e741293efd852c1448a65133, NAME => 'testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRecoveryAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1603): Closing ae5b7764e741293efd852c1448a65133, disabling compactions & flushes 2023-02-09 20:55:44,860 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. after waiting 0 ms 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:44,860 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:44,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1557): Region close journal for ae5b7764e741293efd852c1448a65133: Waiting for close lock at 1675976144860Disabling compacts and flushes for region at 1675976144860Disabling writes for close at 1675976144860Writing region close event to WAL at 1675976144860Closed at 1675976144860 2023-02-09 20:55:44,862 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:55:44,863 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976144863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976144863"}]},"ts":"1675976144863"} 2023-02-09 20:55:44,864 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:55:44,865 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:55:44,865 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976144865"}]},"ts":"1675976144865"} 2023-02-09 20:55:44,866 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLING in hbase:meta 2023-02-09 20:55:44,870 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:55:44,871 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:55:44,871 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:55:44,871 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:55:44,871 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:55:44,871 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, ASSIGN}] 2023-02-09 20:55:44,873 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, ASSIGN 2023-02-09 20:55:44,873 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:45,023 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:55:45,024 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ae5b7764e741293efd852c1448a65133, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,025 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976145024"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976145024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976145024"}]},"ts":"1675976145024"} 2023-02-09 20:55:45,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE; OpenRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:45,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ae5b7764e741293efd852c1448a65133, NAME => 'testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:55:45,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:45,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:45,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,182 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,183 DEBUG [StoreOpener-ae5b7764e741293efd852c1448a65133-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1 2023-02-09 20:55:45,183 DEBUG [StoreOpener-ae5b7764e741293efd852c1448a65133-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1 2023-02-09 20:55:45,184 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae5b7764e741293efd852c1448a65133 columnFamilyName cf1 2023-02-09 20:55:45,184 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] regionserver.HStore(310): Store=ae5b7764e741293efd852c1448a65133/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:45,184 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,185 DEBUG [StoreOpener-ae5b7764e741293efd852c1448a65133-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2 2023-02-09 20:55:45,186 DEBUG [StoreOpener-ae5b7764e741293efd852c1448a65133-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2 2023-02-09 20:55:45,186 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ae5b7764e741293efd852c1448a65133 columnFamilyName cf2 2023-02-09 20:55:45,186 INFO [StoreOpener-ae5b7764e741293efd852c1448a65133-1] regionserver.HStore(310): Store=ae5b7764e741293efd852c1448a65133/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:45,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:45,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:45,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:55:45,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened ae5b7764e741293efd852c1448a65133; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67263285, jitterRate=0.002301052212715149}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:45,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for ae5b7764e741293efd852c1448a65133: Running coprocessor pre-open hook at 1675976145181Writing region info on filesystem at 1675976145181Initializing all the Stores at 1675976145182 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976145182Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976145182Cleaning up temporary data from old regions at 1675976145188 (+6 ms)Cleaning up detritus from prior splits at 1675976145188Running coprocessor post-open hooks at 1675976145191 (+3 ms)Region opened successfully at 1675976145191 2023-02-09 20:55:45,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133., pid=111, masterSystemTime=1675976145178 2023-02-09 20:55:45,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,194 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ae5b7764e741293efd852c1448a65133, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,194 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976145194"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976145194"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976145194"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976145194"}]},"ts":"1675976145194"} 2023-02-09 20:55:45,196 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=110 2023-02-09 20:55:45,196 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=110, state=SUCCESS; OpenRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 in 169 msec 2023-02-09 20:55:45,198 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=109 2023-02-09 20:55:45,198 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=109, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, ASSIGN in 326 msec 2023-02-09 20:55:45,199 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:55:45,199 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976145199"}]},"ts":"1675976145199"} 2023-02-09 20:55:45,200 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLED in hbase:meta 2023-02-09 20:55:45,202 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:55:45,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution in 369 msec 2023-02-09 20:55:45,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:45,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:45,342 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,343 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,343 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:45,343 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:45,343 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,343 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,344 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,344 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,344 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,344 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,345 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,346 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,346 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,346 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:45,347 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:55:45,348 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,350 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,350 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,350 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,350 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,350 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,356 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/f802da4f1c2247618a402d36c901e200 2023-02-09 20:55:45,367 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/98c85b6c0ad041969782f43e9c4e8e93 2023-02-09 20:55:45,370 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/f802da4f1c2247618a402d36c901e200 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 2023-02-09 20:55:45,374 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:45,374 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/98c85b6c0ad041969782f43e9c4e8e93 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 2023-02-09 20:55:45,377 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:45,378 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 31ms, sequenceid=6, compaction requested=false 2023-02-09 20:55:45,378 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,379 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:45,379 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,379 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:45,379 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:45,379 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,379 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,379 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,380 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,380 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,380 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,381 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:45,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70832ff1[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:45,382 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:45,382 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,383 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,383 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,383 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:45,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,384 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:45,385 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,385 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,385 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,387 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,387 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,387 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,388 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,387 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,388 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,388 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:45,390 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:45,390 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:45,390 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:45,390 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,390 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,392 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,393 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,393 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,394 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,395 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,395 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,397 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,398 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:45,398 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,399 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,399 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,399 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,399 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,399 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,436 ERROR [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid a0ea1a13-8490-4cba-a134-da32a3f13324) service to localhost/127.0.0.1:38591 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:55:45,499 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:45,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:45,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:45,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:45,511 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,511 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,511 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:45,511 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:45,512 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,512 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,513 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,513 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,513 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,514 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:45,515 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:45,516 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,516 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:45,516 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:45,517 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,518 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,518 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,518 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,518 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,518 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,524 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/37f9bef11140462ea7927a782b450786 2023-02-09 20:55:45,536 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/6929a2de2ad04f1ebe73f7447f02ca25 2023-02-09 20:55:45,540 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/37f9bef11140462ea7927a782b450786 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 2023-02-09 20:55:45,543 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:45,544 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/6929a2de2ad04f1ebe73f7447f02ca25 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 2023-02-09 20:55:45,547 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:45,548 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 32ms, sequenceid=19, compaction requested=false 2023-02-09 20:55:45,548 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:45,548 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:45,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:45,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,548 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,552 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,552 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,552 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,552 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142c31f9[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:45,554 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:45,554 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,556 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:45,556 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:45,556 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:45,558 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,558 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,558 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,560 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,560 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:45,563 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:45,563 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:45,563 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:45,563 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,563 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,565 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,566 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,568 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,568 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,568 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,571 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:45,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,572 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,672 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:45,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:45,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:45,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:45,685 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:45,685 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:45,686 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,686 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,687 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,687 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,687 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,688 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,688 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:45,690 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,692 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:45,693 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,693 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,693 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:45,693 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:45,695 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,696 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,696 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,696 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,696 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,696 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,707 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/c5ada1db46924493854929cff57ffba4 2023-02-09 20:55:45,729 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/c8169ccc8bd447a2a494d805e3208264 2023-02-09 20:55:45,733 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/c5ada1db46924493854929cff57ffba4 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 2023-02-09 20:55:45,737 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:45,738 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/c8169ccc8bd447a2a494d805e3208264 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 2023-02-09 20:55:45,741 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:45,742 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 49ms, sequenceid=32, compaction requested=true 2023-02-09 20:55:45,742 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:45,742 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,742 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:45,743 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:45,743 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,743 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,743 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,744 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,744 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,745 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,745 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59a1983[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:45,748 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:45,748 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:45,750 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,751 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,751 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:45,751 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,751 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:45,753 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,753 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,753 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,754 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,754 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:45,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:45,758 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:45,759 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:45,759 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:45,759 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,759 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,761 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,762 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,763 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,764 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,766 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,766 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,767 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,774 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,774 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:45,775 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,776 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,876 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:45,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:45,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:45,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:45,891 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:45,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:45,891 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,892 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,893 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,893 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,895 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,895 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,896 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:45,896 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:45,897 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,898 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,898 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,898 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,898 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,905 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/f5f799f04f0246b3824d7fd0adb4155a 2023-02-09 20:55:45,917 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/75cc477e0a404cfebedec100957fbfae 2023-02-09 20:55:45,921 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/f5f799f04f0246b3824d7fd0adb4155a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a 2023-02-09 20:55:45,924 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:45,925 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/75cc477e0a404cfebedec100957fbfae as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae 2023-02-09 20:55:45,928 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:45,929 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 33ms, sequenceid=45, compaction requested=true 2023-02-09 20:55:45,929 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:45,929 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:45,929 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:45,929 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:45,929 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,929 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:45,929 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:45,934 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,934 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,935 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,935 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260d8ec7[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:45,936 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:45,936 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,938 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,938 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:45,938 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:45,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,938 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,939 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:45,939 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,939 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:45,939 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:45,939 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:45,940 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,940 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,940 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,941 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,941 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,941 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,942 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:45,942 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,942 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,942 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:45,945 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:45,945 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:45,945 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:45,945 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:45,945 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,948 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,948 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:45,949 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:45,950 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,950 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,952 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:45,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:45,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:45,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:45,952 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,953 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:45,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,053 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:46,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:46,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:46,067 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:46,067 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,067 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:46,067 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:46,067 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,068 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,069 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,069 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,069 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,071 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,071 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,071 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:46,071 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:46,073 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,073 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,073 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,074 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,075 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,075 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,079 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/1e6c6280d4664871863cf77480698cc4 2023-02-09 20:55:46,089 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/bc99da69f2b54d15803847d7aacb3b39 2023-02-09 20:55:46,093 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/1e6c6280d4664871863cf77480698cc4 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 2023-02-09 20:55:46,096 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:46,097 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/bc99da69f2b54d15803847d7aacb3b39 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 2023-02-09 20:55:46,100 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:46,101 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 30ms, sequenceid=58, compaction requested=true 2023-02-09 20:55:46,101 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:46,101 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:46,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:46,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:46,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,105 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,105 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,106 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,106 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f18c150[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:46,107 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:46,108 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,110 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,111 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,111 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,110 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,111 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:46,111 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,111 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,111 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:46,110 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:46,112 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,112 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,112 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,113 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,113 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:46,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:46,115 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:46,115 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:46,115 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:46,115 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,115 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,117 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,117 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,118 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,118 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,120 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,120 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,120 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,123 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,123 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,123 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,123 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,123 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:46,124 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,125 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,225 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:46,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:46,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-02-09 20:55:46,237 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:46,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-02-09 20:55:46,238 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-02-09 20:55:46,238 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,238 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,239 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,239 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,239 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,239 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,240 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,240 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-02-09 20:55:46,241 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,242 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:46,242 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:46,242 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:46,242 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. started... 2023-02-09 20:55:46,242 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:46,243 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,244 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,244 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,244 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,244 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,244 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,250 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/d9498bbfe4954279ba807db5b4fb0bd8 2023-02-09 20:55:46,265 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/5ab471af3b5b4dd39adfcd071fede69d 2023-02-09 20:55:46,268 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/d9498bbfe4954279ba807db5b4fb0bd8 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 2023-02-09 20:55:46,271 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:46,272 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/5ab471af3b5b4dd39adfcd071fede69d as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d 2023-02-09 20:55:46,275 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:46,276 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 34ms, sequenceid=71, compaction requested=true 2023-02-09 20:55:46,276 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for ae5b7764e741293efd852c1448a65133: 2023-02-09 20:55:46,276 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:46,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:46,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:46,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-02-09 20:55:46,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-02-09 20:55:46,278 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,278 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,279 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,279 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-02-09 20:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ace92b3[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:46,280 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-02-09 20:55:46,280 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:46,282 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-02-09 20:55:46,282 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:46,283 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,283 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,283 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,284 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,284 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-02-09 20:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-02-09 20:55:46,287 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-02-09 20:55:46,287 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:46,287 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:46,287 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-02-09 20:55:46,287 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,290 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,290 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,291 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:46,292 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-02-09 20:55:46,293 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,293 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,298 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-02-09 20:55:46,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:46,298 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,299 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-02-09 20:55:46,399 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:46,660 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-02-09 20:55:46,661 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:46,662 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,663 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,663 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,663 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,664 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:46,664 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:46,664 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:46,912 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 0 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,912 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:46,913 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:46,913 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:46,914 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:46,915 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-02-09 20:55:46,916 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:46,916 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:46,916 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:46,916 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:46,917 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:46,917 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:46,917 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:46,917 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:46,918 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:46,918 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:46,918 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:46,918 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:46,918 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:46,919 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:46,919 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:46,919 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:46,919 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:46,920 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:46,920 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:46,920 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:46,920 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:46,920 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:46,921 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:46,921 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:46,921 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:46,921 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:46,922 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:46,922 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:46,922 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:46,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:46,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:46,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:46,923 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:46,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:46,924 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,924 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:46,924 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,924 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,924 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,925 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-02-09 20:55:46,925 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:46,926 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:46,926 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976176926 2023-02-09 20:55:46,927 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:46,927 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,930 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:46,930 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:46,930 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:46,930 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976206930 2023-02-09 20:55:46,930 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976266930 2023-02-09 20:55:46,930 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-09 20:55:46,930 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:46,932 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:46,932 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:46,932 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:46,932 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:47,180 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 1 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,180 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:47,181 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:47,181 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:47,182 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:47,183 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-02-09 20:55:47,183 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,184 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:47,185 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:47,186 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,187 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,188 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,188 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,188 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:47,188 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,188 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:47,188 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,188 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,188 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,188 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:55:47,189 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:47,189 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:47,190 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976177190 2023-02-09 20:55:47,191 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:47,191 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,193 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,194 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,194 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:47,194 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976207194 2023-02-09 20:55:47,194 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976267194 2023-02-09 20:55:47,194 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-09 20:55:47,194 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:47,196 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN}] 2023-02-09 20:55:47,196 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:47,196 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,196 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:47,196 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:47,444 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 2 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,444 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:47,444 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:47,444 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:47,444 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:47,445 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:47,445 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:47,445 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:47,445 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:47,445 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:47,447 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-02-09 20:55:47,447 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:47,448 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:47,448 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:47,448 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:47,448 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,449 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:47,449 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,449 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,449 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,449 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,450 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,451 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:47,451 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,451 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,451 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,452 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:47,452 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,452 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,452 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,452 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,453 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:47,453 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,453 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,454 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,454 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,454 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,454 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:47,455 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,455 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:47,455 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,455 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,455 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,455 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-02-09 20:55:47,456 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:47,456 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:47,457 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976177457 2023-02-09 20:55:47,458 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:47,458 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,460 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,460 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,460 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:47,461 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976207461 2023-02-09 20:55:47,461 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976267461 2023-02-09 20:55:47,461 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:55:47,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN}] 2023-02-09 20:55:47,461 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:47,463 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:47,463 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=ae5b7764e741293efd852c1448a65133, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:47,464 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976147463"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976147463"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976147463"}]},"ts":"1675976147463"} 2023-02-09 20:55:47,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=113, state=RUNNABLE; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:47,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing ae5b7764e741293efd852c1448a65133, disabling compactions & flushes 2023-02-09 20:55:47,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:47,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:47,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. after waiting 0 ms 2023-02-09 20:55:47,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:47,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing ae5b7764e741293efd852c1448a65133 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:55:47,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/720dbaf4828b43fba94f8bf0bd7c119f 2023-02-09 20:55:47,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/7c6c60847a044c13b30908bd3ea24881 2023-02-09 20:55:47,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf1/720dbaf4828b43fba94f8bf0bd7c119f as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f 2023-02-09 20:55:47,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:47,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/.tmp/cf2/7c6c60847a044c13b30908bd3ea24881 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 2023-02-09 20:55:47,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:47,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 45ms, sequenceid=82, compaction requested=true 2023-02-09 20:55:47,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:55:47,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:47,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. 2023-02-09 20:55:47,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for ae5b7764e741293efd852c1448a65133: Waiting for close lock at 1675976147616Running coprocessor pre-close hooks at 1675976147616Disabling compacts and flushes for region at 1675976147616Disabling writes for close at 1675976147617 (+1 ms)Obtaining lock to block concurrent updates at 1675976147617Preparing flush snapshotting stores in ae5b7764e741293efd852c1448a65133 at 1675976147617Finished memstore snapshotting testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976147617Flushing stores of testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. at 1675976147617Flushing ae5b7764e741293efd852c1448a65133/cf1: creating writer at 1675976147618 (+1 ms)Flushing ae5b7764e741293efd852c1448a65133/cf1: appending metadata at 1675976147620 (+2 ms)Flushing ae5b7764e741293efd852c1448a65133/cf1: closing flushed file at 1675976147620Flushing ae5b7764e741293efd852c1448a65133/cf2: creating writer at 1675976147632 (+12 ms)Flushing ae5b7764e741293efd852c1448a65133/cf2: appending metadata at 1675976147637 (+5 ms)Flushing ae5b7764e741293efd852c1448a65133/cf2: closing flushed file at 1675976147637Flushing ae5b7764e741293efd852c1448a65133/cf1: reopening flushed file at 1675976147653 (+16 ms)Flushing ae5b7764e741293efd852c1448a65133/cf2: reopening flushed file at 1675976147658 (+5 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for ae5b7764e741293efd852c1448a65133 in 45ms, sequenceid=82, compaction requested=true at 1675976147662 (+4 ms)Writing region close event to WAL at 1675976147663 (+1 ms)Running coprocessor post-close hooks at 1675976147666 (+3 ms)Closed at 1675976147667 (+1 ms) 2023-02-09 20:55:47,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,670 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=ae5b7764e741293efd852c1448a65133, regionState=CLOSED 2023-02-09 20:55:47,670 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976147670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976147670"}]},"ts":"1675976147670"} 2023-02-09 20:55:47,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=113 2023-02-09 20:55:47,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 in 206 msec 2023-02-09 20:55:47,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=112 2023-02-09 20:55:47,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN in 212 msec 2023-02-09 20:55:47,677 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:47,677 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:47,677 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:47,711 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 3 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,711 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:47,711 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:47,711 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:47,712 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:47,712 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:47,712 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:47,712 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:47,712 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:47,712 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:47,714 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:47,715 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:47,715 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,715 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:47,715 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,715 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:47,716 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,716 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:47,716 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,716 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:47,716 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,717 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:47,717 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:47,717 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,717 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:47,718 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,719 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:47,720 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,720 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,720 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:47,720 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,721 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:47,721 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,721 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,721 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:47,721 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:47,722 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,722 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,722 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:47,722 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:47,722 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-02-09 20:55:47,723 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:47,723 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:47,724 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976177724 2023-02-09 20:55:47,725 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:47,728 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,729 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,729 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:47,729 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:47,729 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976207729 2023-02-09 20:55:47,729 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976267729 2023-02-09 20:55:47,729 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:47,732 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:47,740 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=ae5b7764e741293efd852c1448a65133, threads=14 2023-02-09 20:55:47,740 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,740 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,740 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,740 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,740 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,741 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,741 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,741 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,741 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,742 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,742 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,742 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,742 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,742 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,908 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,916 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,918 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,921 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,921 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,926 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,931 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,935 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,943 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,943 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,951 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,959 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,959 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,961 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:47,961 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region ae5b7764e741293efd852c1448a65133 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:55:47,990 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:47,990 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:47,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:47,990 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:48,085 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:55:48,229 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 4 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:48,230 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:48,230 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:48,231 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:48,232 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-02-09 20:55:48,232 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:48,232 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:48,232 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,233 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,234 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:48,235 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:48,236 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:48,237 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,237 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,237 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:48,237 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:48,237 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:55:48,238 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:48,238 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:48,239 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976178239 2023-02-09 20:55:48,240 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,240 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:48,243 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:48,243 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:48,243 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:48,243 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976208243 2023-02-09 20:55:48,243 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976268243 2023-02-09 20:55:48,243 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 2 msec 2023-02-09 20:55:48,253 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=ae5b7764e741293efd852c1448a65133, threads=14 2023-02-09 20:55:48,256 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,256 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,257 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,257 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,257 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,257 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,258 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,258 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,258 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,259 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,263 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,263 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,264 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,271 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,363 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,371 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,373 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,376 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,379 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,381 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,392 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,392 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,396 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,777 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,793 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,793 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,793 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,793 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 for region: ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:48,794 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region ae5b7764e741293efd852c1448a65133 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:55:48,823 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:48,827 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:48,828 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:48,828 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:48,828 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,829 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:48,829 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:48,993 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 5 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:48,994 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:48,994 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:48,995 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:48,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-02-09 20:55:48,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:48,996 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:48,996 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:48,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:48,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:48,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:48,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,997 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:48,998 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:48,999 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,000 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,000 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:49,000 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,000 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,000 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,001 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:49,002 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:49,003 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,003 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,003 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,003 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,003 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,003 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-02-09 20:55:49,004 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:49,004 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:49,004 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976179004 2023-02-09 20:55:49,004 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:49,006 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,008 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,008 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,008 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:49,009 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976209009 2023-02-09 20:55:49,009 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976269009 2023-02-09 20:55:49,009 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:49,011 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:49,013 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:49,013 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:49,013 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:49,259 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 6 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,259 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:49,259 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:49,259 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:49,259 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:49,259 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:49,260 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:49,260 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:49,260 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:49,260 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:49,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-02-09 20:55:49,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:49,261 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,262 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,263 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,264 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,265 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:49,266 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:49,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,266 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,267 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,267 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,267 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:55:49,267 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:49,268 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:49,268 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976179268 2023-02-09 20:55:49,268 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:49,270 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,273 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,273 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,273 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:49,273 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976209273 2023-02-09 20:55:49,273 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976269273 2023-02-09 20:55:49,273 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:49,273 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:49,275 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976149275"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1675976149275"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1675976149275"}]},"ts":"1675976149275"} 2023-02-09 20:55:49,275 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976149275"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976149275"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976149275"}]},"ts":"1675976149275"} 2023-02-09 20:55:49,275 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976149275"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976149275"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976149275"}]},"ts":"1675976149275"} 2023-02-09 20:55:49,277 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:49,277 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,277 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:49,277 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:49,523 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 7 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,523 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:49,523 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:49,523 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:49,524 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:49,524 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:49,524 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:49,524 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:49,524 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:49,524 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:49,525 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-02-09 20:55:49,525 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:49,526 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:49,526 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:49,526 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:49,526 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:49,526 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:49,527 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,527 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:49,527 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,527 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,527 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,528 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:49,528 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,528 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,528 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:49,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:49,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,529 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:49,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,530 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:49,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,531 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,532 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,532 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:49,533 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,533 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,533 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,533 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,533 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:49,534 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,534 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,534 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,534 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,535 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-02-09 20:55:49,535 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:49,536 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:49,536 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976179536 2023-02-09 20:55:49,536 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:49,537 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,540 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,540 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,540 INFO [Listener at localhost/45551] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:49,540 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=801747f0d9c820df4e659aa10f0b560f, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:49,540 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=96b439d14200ad59696744407d6b78a9, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:49,540 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:49,540 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976209540 2023-02-09 20:55:49,540 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976269540 2023-02-09 20:55:49,541 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:49,541 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1675976149541"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1675976149541"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1675976149541"}]},"ts":"1675976149541"} 2023-02-09 20:55:49,541 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976149541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976149541"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976149541"}]},"ts":"1675976149541"} 2023-02-09 20:55:49,541 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976149541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976149541"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976149541"}]},"ts":"1675976149541"} 2023-02-09 20:55:49,543 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:49,544 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:49,544 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:49,544 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 8 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:49,791 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:49,791 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:49,792 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:49,793 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11] 2023-02-09 20:55:49,793 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:49,793 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:49,793 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:49,794 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,795 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,796 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,797 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:49,798 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,798 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,798 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,798 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,799 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:49,799 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:55:49,799 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:49,800 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:49,800 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976179800 2023-02-09 20:55:49,800 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:49,801 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,804 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,804 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:49,804 INFO [Listener at localhost/45551] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:49,804 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=801747f0d9c820df4e659aa10f0b560f, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:49,804 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=96b439d14200ad59696744407d6b78a9, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:49,804 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:49,804 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976209804 2023-02-09 20:55:49,804 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976269804 2023-02-09 20:55:49,804 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:49,804 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:49,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN}] 2023-02-09 20:55:49,806 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:49,806 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:49,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:49,806 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:50,054 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 9 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:50,055 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:50,055 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:50,056 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:50,057 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12] 2023-02-09 20:55:50,057 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:50,057 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:50,057 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,058 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:50,059 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,060 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,061 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:50,062 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,062 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,062 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,062 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,062 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-02-09 20:55:50,063 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:50,063 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:50,064 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976180064 2023-02-09 20:55:50,064 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:50,065 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:50,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:50,068 INFO [Listener at localhost/45551] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:50,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=801747f0d9c820df4e659aa10f0b560f, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:50,068 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=96b439d14200ad59696744407d6b78a9, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-02-09 20:55:50,068 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:50,068 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976210068 2023-02-09 20:55:50,068 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976270068 2023-02-09 20:55:50,068 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 4 msec 2023-02-09 20:55:50,069 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN}] 2023-02-09 20:55:50,069 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:50,071 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN 2023-02-09 20:55:50,071 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:50,072 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN 2023-02-09 20:55:50,072 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN; state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:50,222 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:55:50,223 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=96b439d14200ad59696744407d6b78a9, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,224 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150223"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976150223"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976150223"}]},"ts":"1675976150223"} 2023-02-09 20:55:50,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=116, state=RUNNABLE; OpenRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:50,227 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=801747f0d9c820df4e659aa10f0b560f, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,227 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150227"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976150227"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976150227"}]},"ts":"1675976150227"} 2023-02-09 20:55:50,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=118, ppid=115, state=RUNNABLE; OpenRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:50,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 96b439d14200ad59696744407d6b78a9, NAME => 'testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.', STARTKEY => '41', ENDKEY => ''} 2023-02-09 20:55:50,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:50,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:50,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,381 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,381 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1 2023-02-09 20:55:50,381 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1 2023-02-09 20:55:50,382 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96b439d14200ad59696744407d6b78a9 columnFamilyName cf1 2023-02-09 20:55:50,392 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-top 2023-02-09 20:55:50,395 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-top 2023-02-09 20:55:50,397 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-top 2023-02-09 20:55:50,400 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-top 2023-02-09 20:55:50,403 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-top 2023-02-09 20:55:50,405 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-top 2023-02-09 20:55:50,409 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-top 2023-02-09 20:55:50,409 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(310): Store=96b439d14200ad59696744407d6b78a9/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:50,409 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,410 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2 2023-02-09 20:55:50,410 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2 2023-02-09 20:55:50,410 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96b439d14200ad59696744407d6b78a9 columnFamilyName cf2 2023-02-09 20:55:50,421 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-top 2023-02-09 20:55:50,425 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-top 2023-02-09 20:55:50,429 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-top 2023-02-09 20:55:50,432 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-top 2023-02-09 20:55:50,435 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-top 2023-02-09 20:55:50,437 ERROR [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid a0ea1a13-8490-4cba-a134-da32a3f13324) service to localhost/127.0.0.1:38591 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:55:50,438 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-top 2023-02-09 20:55:50,440 DEBUG [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-top 2023-02-09 20:55:50,441 INFO [StoreOpener-96b439d14200ad59696744407d6b78a9-1] regionserver.HStore(310): Store=96b439d14200ad59696744407d6b78a9/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:50,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:50,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 96b439d14200ad59696744407d6b78a9; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62927774, jitterRate=-0.0623030960559845}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:50,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 96b439d14200ad59696744407d6b78a9: Running coprocessor pre-open hook at 1675976150380Writing region info on filesystem at 1675976150380Initializing all the Stores at 1675976150380Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976150380Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976150380Cleaning up temporary data from old regions at 1675976150443 (+63 ms)Cleaning up detritus from prior splits at 1675976150443Running coprocessor post-open hooks at 1675976150446 (+3 ms)Region opened successfully at 1675976150446 2023-02-09 20:55:50,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9., pid=117, masterSystemTime=1675976150376 2023-02-09 20:55:50,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,448 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:50,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:55:50,448 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,448 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): 96b439d14200ad59696744407d6b78a9/cf1 is initiating minor compaction (all files) 2023-02-09 20:55:50,448 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): 96b439d14200ad59696744407d6b78a9/cf2 is initiating minor compaction (all files) 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 96b439d14200ad59696744407d6b78a9/cf1 in testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 96b439d14200ad59696744407d6b78a9/cf2 in testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/.tmp, totalSize=34.9 K 2023-02-09 20:55:50,448 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/.tmp, totalSize=34.9 K 2023-02-09 20:55:50,449 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976145337 2023-02-09 20:55:50,449 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976145337 2023-02-09 20:55:50,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:50,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 801747f0d9c820df4e659aa10f0b560f, NAME => 'testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.', STARTKEY => '', ENDKEY => '41'} 2023-02-09 20:55:50,450 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976145500 2023-02-09 20:55:50,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:50,450 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=96b439d14200ad59696744407d6b78a9, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,450 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976145500 2023-02-09 20:55:50,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150450"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976150450"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976150450"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976150450"}]},"ts":"1675976150450"} 2023-02-09 20:55:50,450 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976145674 2023-02-09 20:55:50,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:50,450 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976145674 2023-02-09 20:55:50,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,451 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976145878 2023-02-09 20:55:50,451 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976145878 2023-02-09 20:55:50,451 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976146055 2023-02-09 20:55:50,452 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976146226 2023-02-09 20:55:50,452 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976146055 2023-02-09 20:55:50,452 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,452 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976146401 2023-02-09 20:55:50,452 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976146226 2023-02-09 20:55:50,453 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1 2023-02-09 20:55:50,453 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1 2023-02-09 20:55:50,453 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976146401 2023-02-09 20:55:50,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=116 2023-02-09 20:55:50,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828 in 226 msec 2023-02-09 20:55:50,454 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 801747f0d9c820df4e659aa10f0b560f columnFamilyName cf1 2023-02-09 20:55:50,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN in 385 msec 2023-02-09 20:55:50,467 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): 96b439d14200ad59696744407d6b78a9#cf2#compaction#108 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:50,467 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96b439d14200ad59696744407d6b78a9#cf1#compaction#109 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:50,469 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-bottom 2023-02-09 20:55:50,473 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-bottom 2023-02-09 20:55:50,481 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-bottom 2023-02-09 20:55:50,483 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-bottom 2023-02-09 20:55:50,491 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-bottom 2023-02-09 20:55:50,496 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-bottom 2023-02-09 20:55:50,497 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/.tmp/cf2/c1abac763e18456593979d092acf1929 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c1abac763e18456593979d092acf1929 2023-02-09 20:55:50,499 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-bottom 2023-02-09 20:55:50,499 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/.tmp/cf1/acf0c69637804deda7440a168bc867ca as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/acf0c69637804deda7440a168bc867ca 2023-02-09 20:55:50,499 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(310): Store=801747f0d9c820df4e659aa10f0b560f/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:50,499 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,500 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2 2023-02-09 20:55:50,500 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2 2023-02-09 20:55:50,501 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 801747f0d9c820df4e659aa10f0b560f columnFamilyName cf2 2023-02-09 20:55:50,502 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96b439d14200ad59696744407d6b78a9/cf2 of 96b439d14200ad59696744407d6b78a9 into c1abac763e18456593979d092acf1929(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:50,502 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 96b439d14200ad59696744407d6b78a9: 2023-02-09 20:55:50,502 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9., storeName=96b439d14200ad59696744407d6b78a9/cf2, priority=9, startTime=1675976150448; duration=0sec 2023-02-09 20:55:50,502 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,504 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96b439d14200ad59696744407d6b78a9/cf1 of 96b439d14200ad59696744407d6b78a9 into acf0c69637804deda7440a168bc867ca(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:50,504 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 96b439d14200ad59696744407d6b78a9: 2023-02-09 20:55:50,504 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9., storeName=96b439d14200ad59696744407d6b78a9/cf1, priority=9, startTime=1675976150447; duration=0sec 2023-02-09 20:55:50,504 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,511 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-bottom 2023-02-09 20:55:50,514 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-bottom 2023-02-09 20:55:50,516 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-bottom 2023-02-09 20:55:50,519 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-bottom 2023-02-09 20:55:50,522 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-bottom 2023-02-09 20:55:50,525 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-bottom 2023-02-09 20:55:50,528 DEBUG [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-bottom 2023-02-09 20:55:50,528 INFO [StoreOpener-801747f0d9c820df4e659aa10f0b560f-1] regionserver.HStore(310): Store=801747f0d9c820df4e659aa10f0b560f/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:50,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,531 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:50,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:50,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 801747f0d9c820df4e659aa10f0b560f; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61517920, jitterRate=-0.08331155776977539}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:50,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 801747f0d9c820df4e659aa10f0b560f: Running coprocessor pre-open hook at 1675976150451Writing region info on filesystem at 1675976150451Initializing all the Stores at 1675976150451Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976150451Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976150452 (+1 ms)Cleaning up temporary data from old regions at 1675976150530 (+78 ms)Cleaning up detritus from prior splits at 1675976150531 (+1 ms)Running coprocessor post-open hooks at 1675976150533 (+2 ms)Region opened successfully at 1675976150533 2023-02-09 20:55:50,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f., pid=118, masterSystemTime=1675976150376 2023-02-09 20:55:50,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,535 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:50,535 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:50,535 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:55:50,535 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,535 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): 801747f0d9c820df4e659aa10f0b560f/cf1 is initiating minor compaction (all files) 2023-02-09 20:55:50,535 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): 801747f0d9c820df4e659aa10f0b560f/cf2 is initiating minor compaction (all files) 2023-02-09 20:55:50,535 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 801747f0d9c820df4e659aa10f0b560f/cf1 in testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,535 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 801747f0d9c820df4e659aa10f0b560f/cf2 in testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,536 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/.tmp, totalSize=34.9 K 2023-02-09 20:55:50,536 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/.tmp, totalSize=34.9 K 2023-02-09 20:55:50,536 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976145337 2023-02-09 20:55:50,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976145337 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976145500 2023-02-09 20:55:50,537 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=801747f0d9c820df4e659aa10f0b560f, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976145500 2023-02-09 20:55:50,537 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150537"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976150537"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976150537"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976150537"}]},"ts":"1675976150537"} 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976145674 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976145674 2023-02-09 20:55:50,537 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976145878 2023-02-09 20:55:50,538 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976145878 2023-02-09 20:55:50,538 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976146055 2023-02-09 20:55:50,538 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976146055 2023-02-09 20:55:50,539 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976146226 2023-02-09 20:55:50,539 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976146226 2023-02-09 20:55:50,539 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976146401 2023-02-09 20:55:50,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=118, resume processing ppid=115 2023-02-09 20:55:50,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828 in 310 msec 2023-02-09 20:55:50,539 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976146401 2023-02-09 20:55:50,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=112 2023-02-09 20:55:50,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN in 471 msec 2023-02-09 20:55:50,542 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-02-09 20:55:50,542 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:50,542 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-02-09 20:55:50,549 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): 801747f0d9c820df4e659aa10f0b560f#cf1#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:50,549 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): 801747f0d9c820df4e659aa10f0b560f#cf2#compaction#111 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:50,561 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/.tmp/cf1/c6c2354696e54fbe9576d27eadf66db7 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c6c2354696e54fbe9576d27eadf66db7 2023-02-09 20:55:50,563 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/.tmp/cf2/bb69d5d645f94bf98250ff647b354577 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bb69d5d645f94bf98250ff647b354577 2023-02-09 20:55:50,565 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 801747f0d9c820df4e659aa10f0b560f/cf1 of 801747f0d9c820df4e659aa10f0b560f into c6c2354696e54fbe9576d27eadf66db7(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:50,565 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 801747f0d9c820df4e659aa10f0b560f: 2023-02-09 20:55:50,565 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f., storeName=801747f0d9c820df4e659aa10f0b560f/cf1, priority=9, startTime=1675976150534; duration=0sec 2023-02-09 20:55:50,565 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,567 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 801747f0d9c820df4e659aa10f0b560f/cf2 of 801747f0d9c820df4e659aa10f0b560f into bb69d5d645f94bf98250ff647b354577(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:50,567 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 801747f0d9c820df4e659aa10f0b560f: 2023-02-09 20:55:50,567 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f., storeName=801747f0d9c820df4e659aa10f0b560f/cf2, priority=9, startTime=1675976150535; duration=0sec 2023-02-09 20:55:50,567 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] procedure.MasterProcedureTestingUtility(454): Restart 10 exec state=pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:55:50,569 ERROR [Listener at localhost/45551] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-02-09 20:55:50,569 INFO [Listener at localhost/45551] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-09 20:55:50,570 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12, 13] 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=115, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[15, 18, 23] 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=118, ppid=115, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[19, 22] 2023-02-09 20:55:50,571 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=116, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[14, 16, 21] 2023-02-09 20:55:50,572 DEBUG [Listener at localhost/45551] store.ProcedureTree(114): Procedure Procedure(pid=117, ppid=116, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[17, 20] 2023-02-09 20:55:50,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-02-09 20:55:50,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-02-09 20:55:50,572 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-02-09 20:55:50,573 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-02-09 20:55:50,573 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,573 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=ec5090c9a8a7e77ccbbcbcb4ed761099, daughterA=81b4f32bb42c8db9e70dbb1fb15261b9, daughterB=11c5f86a4b18d67a5f0febb048d48b9f 2023-02-09 20:55:50,573 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,573 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-02-09 20:55:50,574 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,574 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=65a23bd5b36d215caa9daae544c2807e, daughterA=7f4003ce5139ea977370b6c09c1a1a3b, daughterB=53cdc9bd760aa9b9473ca264cd521151 2023-02-09 20:55:50,574 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,574 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-02-09 20:55:50,574 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,575 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=d14b935178c82ede44119501cb287f81, daughterA=8eee9e859e7e19ef31af324e2d22b2c3, daughterB=b2171a2584fed6219ade2983f2412e19 2023-02-09 20:55:50,575 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=d14b935178c82ede44119501cb287f81, ASSIGN 2023-02-09 20:55:50,575 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,575 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=8a3ca7874d45ec640c0690ef490727ea, daughterA=5e2dbb380e7a15a98ea33055bf2e466c, daughterB=41c1b8256357c0b55c682056898fc851 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=f914af6ac487ff13c2af3eebff56a1b1, daughterA=c61416884e2b4074206df3e8b1ddee75, daughterB=68761db75920bcee7891cf83b0ee94c4 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-02-09 20:55:50,576 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,577 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,577 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRollbackForSplitTableRegionWithReplica, region=50cb6759e917f82973244c041cb8ef94 is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=8b30263dbcee1c353548dc8aaf90953a, daughterA=d53d3a45e9cdc61c4a2f24b114eae9fa, daughterB=402a689856c1be4ca967785518644038 2023-02-09 20:55:50,577 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,577 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-02-09 20:55:50,577 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,578 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,578 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-02-09 20:55:50,578 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,579 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(468): Loading pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 held the lock before restarting, call acquireLock to restore it. 2023-02-09 20:55:50,580 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,580 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,580 INFO [Listener at localhost/45551] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure ae5b7764e741293efd852c1448a65133, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 DEBUG [Listener at localhost/45551] procedure2.Procedure(946): pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828 didn't hold the lock before restarting, skip acquiring lock. 2023-02-09 20:55:50,580 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-02-09 20:55:50,581 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-02-09 20:55:50,581 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828, table=hbase:meta, region=1588230740 2023-02-09 20:55:50,581 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1675976180581 2023-02-09 20:55:50,583 DEBUG [Listener at localhost/45551] assignment.AssignmentManager(1519): Joining cluster... 2023-02-09 20:55:50,583 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:50,586 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=9a9aa832e601f0693eaa144f8209e596, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:50,586 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=ae5b7764e741293efd852c1448a65133, regionState=CLOSED, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=2 2023-02-09 20:55:50,586 INFO [Listener at localhost/45551] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase9.apache.org,43365,1675976048828, table=testRecoveryAndDoubleExecution, region=ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:50,586 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=801747f0d9c820df4e659aa10f0b560f, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=86 2023-02-09 20:55:50,586 DEBUG [Listener at localhost/45551] assignment.RegionStateStore(147): Load hbase:meta entry region=96b439d14200ad59696744407d6b78a9, regionState=OPEN, lastHost=jenkins-hbase9.apache.org,43365,1675976048828, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828, openSeqNum=86 2023-02-09 20:55:50,587 INFO [Listener at localhost/45551] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-09 20:55:50,587 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1675976210587 2023-02-09 20:55:50,587 INFO [Listener at localhost/45551] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1675976270587 2023-02-09 20:55:50,587 INFO [Listener at localhost/45551] assignment.AssignmentManager(1538): Joined the cluster in 3 msec 2023-02-09 20:55:50,587 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-02-09 20:55:50,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, state=SUCCESS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=ae5b7764e741293efd852c1448a65133, daughterA=801747f0d9c820df4e659aa10f0b560f, daughterB=96b439d14200ad59696744407d6b78a9 in 3.9260 sec 2023-02-09 20:55:50,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43365] ipc.CallRunner(144): callId: 856 service: ClientService methodName: Scan size: 154 connection: 172.31.2.10:56516 deadline: 1675976210838, exception=org.apache.hadoop.hbase.NotServingRegionException: testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133. is not online on jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,949 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:55:50,957 INFO [Listener at localhost/45551] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-09 20:55:50,958 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:55:50,968 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:50,969 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testRecoveryAndDoubleExecution 2023-02-09 20:55:50,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testRecoveryAndDoubleExecution 2023-02-09 20:55:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-09 20:55:50,972 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976150972"}]},"ts":"1675976150972"} 2023-02-09 20:55:50,973 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLING in hbase:meta 2023-02-09 20:55:50,975 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRecoveryAndDoubleExecution to state=DISABLING 2023-02-09 20:55:50,975 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, UNASSIGN}, {pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, UNASSIGN}] 2023-02-09 20:55:50,976 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, UNASSIGN 2023-02-09 20:55:50,976 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=96b439d14200ad59696744407d6b78a9, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,976 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150976"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976150976"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976150976"}]},"ts":"1675976150976"} 2023-02-09 20:55:50,977 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE; CloseRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:50,979 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, UNASSIGN 2023-02-09 20:55:50,979 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=801747f0d9c820df4e659aa10f0b560f, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:50,979 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976150979"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976150979"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976150979"}]},"ts":"1675976150979"} 2023-02-09 20:55:50,980 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=123, ppid=120, state=RUNNABLE; CloseRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:51,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-09 20:55:51,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:51,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 96b439d14200ad59696744407d6b78a9, disabling compactions & flushes 2023-02-09 20:55:51,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:51,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:51,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. after waiting 0 ms 2023-02-09 20:55:51,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:51,130 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-top] to archive 2023-02-09 20:55:51,131 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:51,132 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,133 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,134 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,136 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,137 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,138 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,139 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,141 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-top] to archive 2023-02-09 20:55:51,141 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:51,143 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,144 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,145 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,146 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,147 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,148 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,149 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:55:51,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:51,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9. 2023-02-09 20:55:51,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 96b439d14200ad59696744407d6b78a9: Waiting for close lock at 1675976151129Running coprocessor pre-close hooks at 1675976151129Disabling compacts and flushes for region at 1675976151129Disabling writes for close at 1675976151129Writing region close event to WAL at 1675976151150 (+21 ms)Running coprocessor post-close hooks at 1675976151153 (+3 ms)Closed at 1675976151153 2023-02-09 20:55:51,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:51,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:51,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 801747f0d9c820df4e659aa10f0b560f, disabling compactions & flushes 2023-02-09 20:55:51,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:51,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:51,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. after waiting 0 ms 2023-02-09 20:55:51,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:51,156 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=96b439d14200ad59696744407d6b78a9, regionState=CLOSED 2023-02-09 20:55:51,156 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976151156"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976151156"}]},"ts":"1675976151156"} 2023-02-09 20:55:51,157 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f-bottom] to archive 2023-02-09 20:55:51,157 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:51,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=121 2023-02-09 20:55:51,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=121, state=SUCCESS; CloseRegionProcedure 96b439d14200ad59696744407d6b78a9, server=jenkins-hbase9.apache.org,43365,1675976048828 in 180 msec 2023-02-09 20:55:51,159 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=96b439d14200ad59696744407d6b78a9, UNASSIGN in 184 msec 2023-02-09 20:55:51,160 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,161 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,162 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,163 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,164 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,165 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,166 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881-bottom] to archive 2023-02-09 20:55:51,167 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:51,168 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,169 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,169 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,170 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,171 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,172 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,173 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,176 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:55:51,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:51,177 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f. 2023-02-09 20:55:51,177 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 801747f0d9c820df4e659aa10f0b560f: Waiting for close lock at 1675976151156Running coprocessor pre-close hooks at 1675976151156Disabling compacts and flushes for region at 1675976151156Disabling writes for close at 1675976151156Writing region close event to WAL at 1675976151174 (+18 ms)Running coprocessor post-close hooks at 1675976151177 (+3 ms)Closed at 1675976151177 2023-02-09 20:55:51,179 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:51,179 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=801747f0d9c820df4e659aa10f0b560f, regionState=CLOSED 2023-02-09 20:55:51,180 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1675976151179"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976151179"}]},"ts":"1675976151179"} 2023-02-09 20:55:51,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-02-09 20:55:51,182 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; CloseRegionProcedure 801747f0d9c820df4e659aa10f0b560f, server=jenkins-hbase9.apache.org,43365,1675976048828 in 201 msec 2023-02-09 20:55:51,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=120, resume processing ppid=119 2023-02-09 20:55:51,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=801747f0d9c820df4e659aa10f0b560f, UNASSIGN in 208 msec 2023-02-09 20:55:51,185 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976151184"}]},"ts":"1675976151184"} 2023-02-09 20:55:51,186 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLED in hbase:meta 2023-02-09 20:55:51,188 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRecoveryAndDoubleExecution to state=DISABLED 2023-02-09 20:55:51,196 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; DisableTableProcedure table=testRecoveryAndDoubleExecution in 225 msec 2023-02-09 20:55:51,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-09 20:55:51,274 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRecoveryAndDoubleExecution, procId: 119 completed 2023-02-09 20:55:51,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testRecoveryAndDoubleExecution 2023-02-09 20:55:51,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,277 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,277 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-02-09 20:55:51,282 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,282 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:51,282 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:51,284 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/recovered.edits] 2023-02-09 20:55:51,284 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/recovered.edits] 2023-02-09 20:55:51,284 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/recovered.edits] 2023-02-09 20:55:51,287 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/.regioninfo to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/.regioninfo 2023-02-09 20:55:51,287 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/.regioninfo to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/.regioninfo 2023-02-09 20:55:51,287 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/1e6c6280d4664871863cf77480698cc4 2023-02-09 20:55:51,288 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/37f9bef11140462ea7927a782b450786 2023-02-09 20:55:51,289 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/720dbaf4828b43fba94f8bf0bd7c119f 2023-02-09 20:55:51,289 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,290 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/1e6c6280d4664871863cf77480698cc4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,290 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/c5ada1db46924493854929cff57ffba4 2023-02-09 20:55:51,290 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,291 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/37f9bef11140462ea7927a782b450786.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,292 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,292 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/d9498bbfe4954279ba807db5b4fb0bd8 2023-02-09 20:55:51,293 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/720dbaf4828b43fba94f8bf0bd7c119f.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,293 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,293 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f5f799f04f0246b3824d7fd0adb4155a 2023-02-09 20:55:51,294 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/c5ada1db46924493854929cff57ffba4.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,294 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,294 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf1/f802da4f1c2247618a402d36c901e200 2023-02-09 20:55:51,295 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/d9498bbfe4954279ba807db5b4fb0bd8.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,295 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,297 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/f5f799f04f0246b3824d7fd0adb4155a.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,297 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,298 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/5ab471af3b5b4dd39adfcd071fede69d 2023-02-09 20:55:51,298 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf1/f802da4f1c2247618a402d36c901e200.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,299 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/6929a2de2ad04f1ebe73f7447f02ca25 2023-02-09 20:55:51,300 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,300 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/75cc477e0a404cfebedec100957fbfae 2023-02-09 20:55:51,301 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,301 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/5ab471af3b5b4dd39adfcd071fede69d.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,302 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/7c6c60847a044c13b30908bd3ea24881 2023-02-09 20:55:51,302 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/6929a2de2ad04f1ebe73f7447f02ca25.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,302 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,303 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/98c85b6c0ad041969782f43e9c4e8e93 2023-02-09 20:55:51,304 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/75cc477e0a404cfebedec100957fbfae.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,304 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,304 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/bc99da69f2b54d15803847d7aacb3b39 2023-02-09 20:55:51,305 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/7c6c60847a044c13b30908bd3ea24881.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,305 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,306 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/cf2/c8169ccc8bd447a2a494d805e3208264 2023-02-09 20:55:51,306 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/98c85b6c0ad041969782f43e9c4e8e93.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,306 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,308 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/bc99da69f2b54d15803847d7aacb3b39.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,308 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/96b439d14200ad59696744407d6b78a9/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,309 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133/recovered.edits/85.seqid 2023-02-09 20:55:51,309 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/801747f0d9c820df4e659aa10f0b560f/cf2/c8169ccc8bd447a2a494d805e3208264.ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,309 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/ae5b7764e741293efd852c1448a65133 2023-02-09 20:55:51,310 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/acf0c69637804deda7440a168bc867ca to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf1/acf0c69637804deda7440a168bc867ca 2023-02-09 20:55:51,311 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c6c2354696e54fbe9576d27eadf66db7 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf1/c6c2354696e54fbe9576d27eadf66db7 2023-02-09 20:55:51,312 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c1abac763e18456593979d092acf1929 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/cf2/c1abac763e18456593979d092acf1929 2023-02-09 20:55:51,314 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bb69d5d645f94bf98250ff647b354577 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/cf2/bb69d5d645f94bf98250ff647b354577 2023-02-09 20:55:51,315 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9/recovered.edits/90.seqid 2023-02-09 20:55:51,315 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/96b439d14200ad59696744407d6b78a9 2023-02-09 20:55:51,316 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f/recovered.edits/90.seqid 2023-02-09 20:55:51,317 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testRecoveryAndDoubleExecution/801747f0d9c820df4e659aa10f0b560f 2023-02-09 20:55:51,317 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-02-09 20:55:51,319 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,320 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testRecoveryAndDoubleExecution from hbase:meta 2023-02-09 20:55:51,322 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRecoveryAndDoubleExecution' descriptor. 2023-02-09 20:55:51,322 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,322 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRecoveryAndDoubleExecution' from region states. 2023-02-09 20:55:51,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976151322"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:51,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976151322"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:51,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976151322"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:51,324 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:55:51,324 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ae5b7764e741293efd852c1448a65133, NAME => 'testRecoveryAndDoubleExecution,,1675976144833.ae5b7764e741293efd852c1448a65133.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => 801747f0d9c820df4e659aa10f0b560f, NAME => 'testRecoveryAndDoubleExecution,,1675976146661.801747f0d9c820df4e659aa10f0b560f.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 96b439d14200ad59696744407d6b78a9, NAME => 'testRecoveryAndDoubleExecution,41,1675976146661.96b439d14200ad59696744407d6b78a9.', STARTKEY => '41', ENDKEY => ''}] 2023-02-09 20:55:51,324 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRecoveryAndDoubleExecution' as deleted. 2023-02-09 20:55:51,324 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976151324"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:51,325 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRecoveryAndDoubleExecution state from META 2023-02-09 20:55:51,327 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-02-09 20:55:51,328 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testRecoveryAndDoubleExecution in 52 msec 2023-02-09 20:55:51,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-02-09 20:55:51,379 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRecoveryAndDoubleExecution, procId: 124 completed 2023-02-09 20:55:51,393 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=376 (was 368) - Thread LEAK? -, OpenFileDescriptor=829 (was 740) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=129 (was 140), ProcessCount=168 (was 168), AvailableMemoryMB=10495 (was 10526) 2023-02-09 20:55:51,406 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=376, OpenFileDescriptor=829, MaxFileDescriptor=60000, SystemLoadAverage=129, ProcessCount=168, AvailableMemoryMB=10494 2023-02-09 20:55:51,406 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:51,409 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:55:51,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:55:51,411 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,412 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:55:51,412 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:55:51,414 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,414 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f empty. 2023-02-09 20:55:51,414 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-02-09 20:55:51,422 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/.tabledesc/.tableinfo.0000000001 2023-02-09 20:55:51,423 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(7671): creating {ENCODED => eda24b137d6b92f849cd269a2ce19e3f, NAME => 'testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionEmptyDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1603): Closing eda24b137d6b92f849cd269a2ce19e3f, disabling compactions & flushes 2023-02-09 20:55:51,430 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. after waiting 0 ms 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,430 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,430 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1557): Region close journal for eda24b137d6b92f849cd269a2ce19e3f: Waiting for close lock at 1675976151430Disabling compacts and flushes for region at 1675976151430Disabling writes for close at 1675976151430Writing region close event to WAL at 1675976151430Closed at 1675976151430 2023-02-09 20:55:51,432 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:55:51,432 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976151432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976151432"}]},"ts":"1675976151432"} 2023-02-09 20:55:51,433 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:55:51,434 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:55:51,434 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976151434"}]},"ts":"1675976151434"} 2023-02-09 20:55:51,435 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLING in hbase:meta 2023-02-09 20:55:51,438 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:55:51,439 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:55:51,439 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:55:51,439 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:55:51,439 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:55:51,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, ASSIGN}] 2023-02-09 20:55:51,440 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, ASSIGN 2023-02-09 20:55:51,440 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:51,441 ERROR [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid 5ff62a41-196c-4396-be17-ac730c5a50b4) service to localhost/127.0.0.1:38591 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:55:51,591 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:55:51,592 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=eda24b137d6b92f849cd269a2ce19e3f, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,592 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976151592"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976151592"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976151592"}]},"ts":"1675976151592"} 2023-02-09 20:55:51,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure eda24b137d6b92f849cd269a2ce19e3f, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:51,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => eda24b137d6b92f849cd269a2ce19e3f, NAME => 'testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:55:51,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:51,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:51,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,749 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,750 DEBUG [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1 2023-02-09 20:55:51,750 DEBUG [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1 2023-02-09 20:55:51,750 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eda24b137d6b92f849cd269a2ce19e3f columnFamilyName cf1 2023-02-09 20:55:51,751 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] regionserver.HStore(310): Store=eda24b137d6b92f849cd269a2ce19e3f/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:51,751 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,752 DEBUG [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2 2023-02-09 20:55:51,752 DEBUG [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2 2023-02-09 20:55:51,752 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region eda24b137d6b92f849cd269a2ce19e3f columnFamilyName cf2 2023-02-09 20:55:51,752 INFO [StoreOpener-eda24b137d6b92f849cd269a2ce19e3f-1] regionserver.HStore(310): Store=eda24b137d6b92f849cd269a2ce19e3f/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:51,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,753 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,754 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:51,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:51,756 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:55:51,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened eda24b137d6b92f849cd269a2ce19e3f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60365367, jitterRate=-0.10048593580722809}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:51,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for eda24b137d6b92f849cd269a2ce19e3f: Running coprocessor pre-open hook at 1675976151748Writing region info on filesystem at 1675976151748Initializing all the Stores at 1675976151749 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976151749Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976151749Cleaning up temporary data from old regions at 1675976151754 (+5 ms)Cleaning up detritus from prior splits at 1675976151754Running coprocessor post-open hooks at 1675976151757 (+3 ms)Region opened successfully at 1675976151757 2023-02-09 20:55:51,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f., pid=127, masterSystemTime=1675976151745 2023-02-09 20:55:51,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,759 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,759 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=eda24b137d6b92f849cd269a2ce19e3f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,759 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976151759"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976151759"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976151759"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976151759"}]},"ts":"1675976151759"} 2023-02-09 20:55:51,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-09 20:55:51,762 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure eda24b137d6b92f849cd269a2ce19e3f, server=jenkins-hbase9.apache.org,43365,1675976048828 in 166 msec 2023-02-09 20:55:51,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=125 2023-02-09 20:55:51,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, ASSIGN in 323 msec 2023-02-09 20:55:51,764 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:55:51,764 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976151764"}]},"ts":"1675976151764"} 2023-02-09 20:55:51,764 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLED in hbase:meta 2023-02-09 20:55:51,766 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:55:51,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionEmptyDaughter in 356 msec 2023-02-09 20:55:51,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,919 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:51,919 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:51,919 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:51,919 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:51,919 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,919 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,923 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,923 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,924 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,924 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:51,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:51,925 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:51,925 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:51,925 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:51,926 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:55:51,926 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:51,927 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,928 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,928 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,928 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:51,929 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,929 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:51,936 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/78f0467acee341509b2723b1c92e2e26 2023-02-09 20:55:51,949 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/37642d2ecf1f4ed59863121ceb8e65b0 2023-02-09 20:55:51,952 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/78f0467acee341509b2723b1c92e2e26 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26 2023-02-09 20:55:51,956 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:51,956 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/37642d2ecf1f4ed59863121ceb8e65b0 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0 2023-02-09 20:55:51,959 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:51,960 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 34ms, sequenceid=6, compaction requested=false 2023-02-09 20:55:51,960 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:51,960 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:51,960 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:51,960 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:51,960 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:51,960 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,960 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:51,960 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:51,962 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,962 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:51,962 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,962 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:51,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:51,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f4dc268[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:51,964 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:51,964 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,966 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:51,966 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:51,968 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:51,968 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:51,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:51,971 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:51,971 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:51,971 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:51,971 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,971 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,972 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:51,973 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:51,974 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,975 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,975 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:51,976 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:51,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:51,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:51,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,984 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:51,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,984 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:51,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:51,985 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,085 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:52,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:52,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:52,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:52,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,097 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:52,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:52,097 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,098 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,099 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,099 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,099 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,100 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,100 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,100 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,100 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:52,101 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:52,101 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,101 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:52,101 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:52,102 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,103 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,103 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,103 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,103 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,103 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,109 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/95465d9994ed4cecb206f4d7dfd0e182 2023-02-09 20:55:52,117 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/c068cac6558c434289b47c40f623b0c8 2023-02-09 20:55:52,120 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/95465d9994ed4cecb206f4d7dfd0e182 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182 2023-02-09 20:55:52,124 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:52,124 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/c068cac6558c434289b47c40f623b0c8 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8 2023-02-09 20:55:52,127 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:52,127 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 26ms, sequenceid=19, compaction requested=false 2023-02-09 20:55:52,128 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:52,128 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,128 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:52,128 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:52,128 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,128 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,128 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,131 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,131 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,132 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,132 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:52,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25fc12f0[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:52,135 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:52,135 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,136 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,136 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,136 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:52,137 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:52,139 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,140 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:52,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:52,147 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:52,147 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:52,147 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:52,147 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,147 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:52,149 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,149 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,149 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,149 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,149 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,150 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,151 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,151 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,153 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,154 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,154 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,157 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,157 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:52,158 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,158 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,159 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,163 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,163 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,259 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:52,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,272 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,272 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,272 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:52,272 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:52,272 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,273 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,274 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,274 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,274 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,277 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,277 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,276 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:52,277 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:52,277 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:52,277 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,277 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,277 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:52,277 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,278 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,280 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,280 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,280 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,280 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,280 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,289 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/8962ff4db13c40698c6932eaee26e8ea 2023-02-09 20:55:52,699 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/2d3b3da6032846fb86fba36fee5a5b1e 2023-02-09 20:55:52,703 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/8962ff4db13c40698c6932eaee26e8ea as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea 2023-02-09 20:55:52,706 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:52,706 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/2d3b3da6032846fb86fba36fee5a5b1e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e 2023-02-09 20:55:52,709 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:52,710 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 433ms, sequenceid=32, compaction requested=true 2023-02-09 20:55:52,710 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:52,710 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,710 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:52,710 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:52,710 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,710 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,710 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,712 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,712 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:52,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c67455e[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:52,714 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:52,714 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,715 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,715 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:52,715 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:52,725 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,725 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,725 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,725 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,726 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:52,728 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:52,729 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:52,729 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:52,729 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,729 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,730 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,731 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,733 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,733 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,733 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,736 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,736 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,737 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,837 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:52,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:52,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:52,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,849 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,849 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,849 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:52,849 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:52,849 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,849 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,850 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,850 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,850 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,850 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,851 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,851 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:52,853 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,852 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,853 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:52,853 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:52,853 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,853 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:52,857 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,858 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,858 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,858 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,858 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,858 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,861 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/8cd232e7eba44fcba61432d1936a4575 2023-02-09 20:55:52,870 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/59d991d1fcd841f0bd3c6d13f9076641 2023-02-09 20:55:52,874 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/8cd232e7eba44fcba61432d1936a4575 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575 2023-02-09 20:55:52,876 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:52,877 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/59d991d1fcd841f0bd3c6d13f9076641 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641 2023-02-09 20:55:52,880 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:52,881 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 28ms, sequenceid=45, compaction requested=true 2023-02-09 20:55:52,881 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:52,881 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:52,881 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:52,881 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:52,881 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,881 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:52,881 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:52,882 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,882 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,883 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,883 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6272016b[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:52,884 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:52,884 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:52,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,886 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:52,886 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:52,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,889 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,890 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,890 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,890 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:52,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:52,893 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:52,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:52,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:52,893 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,893 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:52,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,894 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,895 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,895 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,895 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,896 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,897 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:52,897 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,898 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:52,905 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:52,906 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,006 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:53,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:53,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:53,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,017 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:53,017 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,017 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:53,017 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:53,018 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,018 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,019 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,019 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,019 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,019 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:53,019 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,020 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:53,021 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:53,021 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,021 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:53,021 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:53,022 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,022 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,022 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,028 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,029 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,029 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,029 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/a2b2c88ece9948d58e126ddf9c5430d2 2023-02-09 20:55:53,038 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/4f0051263d9e44b4bee77780a98cb22e 2023-02-09 20:55:53,041 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/a2b2c88ece9948d58e126ddf9c5430d2 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2 2023-02-09 20:55:53,044 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:53,045 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/4f0051263d9e44b4bee77780a98cb22e as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e 2023-02-09 20:55:53,048 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:53,048 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 27ms, sequenceid=58, compaction requested=true 2023-02-09 20:55:53,048 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:53,048 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,049 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:53,049 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:53,049 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,049 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,049 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,050 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,050 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,051 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,051 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63adc1cf[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:53,052 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:53,052 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,053 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,053 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,053 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,054 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,055 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,057 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,057 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,057 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:53,060 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:53,060 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:53,060 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:53,060 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,060 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:53,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,062 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,062 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,064 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,064 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,065 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:53,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,067 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:53,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,067 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,068 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,168 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:55:53,168 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:53,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:53,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,179 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:53,179 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,179 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-02-09 20:55:53,179 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-02-09 20:55:53,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,180 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,181 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,182 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,182 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-02-09 20:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:53,184 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:53,184 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,184 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. started... 2023-02-09 20:55:53,185 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:53,187 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,187 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,187 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,188 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,188 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,188 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,196 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/1f1834684b004d93ac1a6cc81f2d6a24 2023-02-09 20:55:53,609 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/b99447302f264b1582f1f32bc9e1bc1a 2023-02-09 20:55:53,613 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/1f1834684b004d93ac1a6cc81f2d6a24 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24 2023-02-09 20:55:53,615 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:53,616 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/b99447302f264b1582f1f32bc9e1bc1a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a 2023-02-09 20:55:53,619 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:53,620 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 435ms, sequenceid=71, compaction requested=true 2023-02-09 20:55:53,620 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for eda24b137d6b92f849cd269a2ce19e3f: 2023-02-09 20:55:53,620 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,620 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:53,620 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:53,620 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,620 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-02-09 20:55:53,620 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-02-09 20:55:53,622 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,622 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,622 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,622 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-02-09 20:55:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260334d8[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:53,624 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-02-09 20:55:53,624 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,628 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,628 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,628 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-02-09 20:55:53,629 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,629 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:53,629 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:53,630 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,630 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,630 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-02-09 20:55:53,631 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,631 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-02-09 20:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-02-09 20:55:53,634 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-02-09 20:55:53,634 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:53,634 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:53,634 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,634 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,635 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,636 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,636 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,636 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,638 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:53,638 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,638 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,641 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-02-09 20:55:53,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:53,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,642 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,643 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-02-09 20:55:53,743 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:53,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:53,750 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,751 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=eda24b137d6b92f849cd269a2ce19e3f, daughterA=96a50bc53ff96fbc6fcb8cf6c4a41083, daughterB=d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:53,752 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=eda24b137d6b92f849cd269a2ce19e3f, daughterA=96a50bc53ff96fbc6fcb8cf6c4a41083, daughterB=d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:53,752 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=eda24b137d6b92f849cd269a2ce19e3f, daughterA=96a50bc53ff96fbc6fcb8cf6c4a41083, daughterB=d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:53,752 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=eda24b137d6b92f849cd269a2ce19e3f, daughterA=96a50bc53ff96fbc6fcb8cf6c4a41083, daughterB=d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:53,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, UNASSIGN}] 2023-02-09 20:55:53,754 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, UNASSIGN 2023-02-09 20:55:53,754 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=eda24b137d6b92f849cd269a2ce19e3f, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:53,754 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976153754"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976153754"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976153754"}]},"ts":"1675976153754"} 2023-02-09 20:55:53,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=129, state=RUNNABLE; CloseRegionProcedure eda24b137d6b92f849cd269a2ce19e3f, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:53,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:53,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing eda24b137d6b92f849cd269a2ce19e3f, disabling compactions & flushes 2023-02-09 20:55:53,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. after waiting 0 ms 2023-02-09 20:55:53,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:53,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing eda24b137d6b92f849cd269a2ce19e3f 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:55:53,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/df5b17ed570d41c99d11fa2c09f3e283 2023-02-09 20:55:54,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/7b2833977cf546fabca7510f7c13aff6 2023-02-09 20:55:54,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf1/df5b17ed570d41c99d11fa2c09f3e283 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283 2023-02-09 20:55:54,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:54,344 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.tmp/cf2/7b2833977cf546fabca7510f7c13aff6 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6 2023-02-09 20:55:54,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:54,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 442ms, sequenceid=82, compaction requested=true 2023-02-09 20:55:54,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:55:54,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:54,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. 2023-02-09 20:55:54,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for eda24b137d6b92f849cd269a2ce19e3f: Waiting for close lock at 1675976153907Running coprocessor pre-close hooks at 1675976153907Disabling compacts and flushes for region at 1675976153907Disabling writes for close at 1675976153907Obtaining lock to block concurrent updates at 1675976153907Preparing flush snapshotting stores in eda24b137d6b92f849cd269a2ce19e3f at 1675976153907Finished memstore snapshotting testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976153908 (+1 ms)Flushing stores of testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. at 1675976153908Flushing eda24b137d6b92f849cd269a2ce19e3f/cf1: creating writer at 1675976153910 (+2 ms)Flushing eda24b137d6b92f849cd269a2ce19e3f/cf1: appending metadata at 1675976153912 (+2 ms)Flushing eda24b137d6b92f849cd269a2ce19e3f/cf1: closing flushed file at 1675976153912Flushing eda24b137d6b92f849cd269a2ce19e3f/cf2: creating writer at 1675976153923 (+11 ms)Flushing eda24b137d6b92f849cd269a2ce19e3f/cf2: appending metadata at 1675976153925 (+2 ms)Flushing eda24b137d6b92f849cd269a2ce19e3f/cf2: closing flushed file at 1675976153925Flushing eda24b137d6b92f849cd269a2ce19e3f/cf1: reopening flushed file at 1675976154341 (+416 ms)Flushing eda24b137d6b92f849cd269a2ce19e3f/cf2: reopening flushed file at 1675976154345 (+4 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for eda24b137d6b92f849cd269a2ce19e3f in 442ms, sequenceid=82, compaction requested=true at 1675976154349 (+4 ms)Writing region close event to WAL at 1675976154350 (+1 ms)Running coprocessor post-close hooks at 1675976154352 (+2 ms)Closed at 1675976154353 (+1 ms) 2023-02-09 20:55:54,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,355 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=eda24b137d6b92f849cd269a2ce19e3f, regionState=CLOSED 2023-02-09 20:55:54,355 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976154355"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976154355"}]},"ts":"1675976154355"} 2023-02-09 20:55:54,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=129 2023-02-09 20:55:54,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=129, state=SUCCESS; CloseRegionProcedure eda24b137d6b92f849cd269a2ce19e3f, server=jenkins-hbase9.apache.org,43365,1675976048828 in 601 msec 2023-02-09 20:55:54,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=128 2023-02-09 20:55:54,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=eda24b137d6b92f849cd269a2ce19e3f, UNASSIGN in 604 msec 2023-02-09 20:55:54,367 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=128 splitting 14 storefiles, region=eda24b137d6b92f849cd269a2ce19e3f, threads=14 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,367 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,368 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,371 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e, top=false 2023-02-09 20:55:54,371 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a, top=false 2023-02-09 20:55:54,378 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26, top=false 2023-02-09 20:55:54,381 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6, top=false 2023-02-09 20:55:54,381 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0, top=false 2023-02-09 20:55:54,381 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575, top=false 2023-02-09 20:55:54,381 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2, top=false 2023-02-09 20:55:54,382 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e, top=false 2023-02-09 20:55:54,383 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea, top=false 2023-02-09 20:55:54,383 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641, top=false 2023-02-09 20:55:54,384 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283, top=false 2023-02-09 20:55:54,384 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8, top=false 2023-02-09 20:55:54,385 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182, top=false 2023-02-09 20:55:54,388 DEBUG [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24, top=false 2023-02-09 20:55:54,395 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,398 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,407 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,407 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,411 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,411 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,419 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,419 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,419 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,419 INFO [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,420 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,419 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,420 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,420 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,421 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283 for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,424 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,427 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,428 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/.splits/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea for child: 96a50bc53ff96fbc6fcb8cf6c4a41083, parent: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,430 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,430 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,431 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,431 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,431 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,431 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,432 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,432 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8 for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,432 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,433 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea for region: eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:54,433 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=128 split storefiles for region eda24b137d6b92f849cd269a2ce19e3f Daughter A: 14 storefiles, Daughter B: 0 storefiles. 2023-02-09 20:55:54,860 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:54,862 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:54,863 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1675976154863"},{"qualifier":"splitA","vlen":69,"tag":[],"timestamp":"1675976154863"},{"qualifier":"splitB","vlen":69,"tag":[],"timestamp":"1675976154863"}]},"ts":"1675976154863"} 2023-02-09 20:55:54,863 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976154863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976154863"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976154863"}]},"ts":"1675976154863"} 2023-02-09 20:55:54,863 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976154863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976154863"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976154863"}]},"ts":"1675976154863"} 2023-02-09 20:55:54,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, ASSIGN}, {pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, ASSIGN}] 2023-02-09 20:55:54,866 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, ASSIGN 2023-02-09 20:55:54,867 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:54,867 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, ASSIGN 2023-02-09 20:55:54,867 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:55,017 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:55:55,019 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=96a50bc53ff96fbc6fcb8cf6c4a41083, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,019 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155019"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976155019"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976155019"}]},"ts":"1675976155019"} 2023-02-09 20:55:55,020 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=133, ppid=131, state=RUNNABLE; OpenRegionProcedure 96a50bc53ff96fbc6fcb8cf6c4a41083, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:55,022 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=d2502d975b11df7ff327f1cde3bb1527, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,022 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155022"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976155022"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976155022"}]},"ts":"1675976155022"} 2023-02-09 20:55:55,023 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=132, state=RUNNABLE; OpenRegionProcedure d2502d975b11df7ff327f1cde3bb1527, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:55,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 96a50bc53ff96fbc6fcb8cf6c4a41083, NAME => 'testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.', STARTKEY => '', ENDKEY => '71'} 2023-02-09 20:55:55,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:55,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:55,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,176 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,177 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1 2023-02-09 20:55:55,177 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1 2023-02-09 20:55:55,178 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96a50bc53ff96fbc6fcb8cf6c4a41083 columnFamilyName cf1 2023-02-09 20:55:55,183 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24 2023-02-09 20:55:55,185 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26 2023-02-09 20:55:55,187 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea 2023-02-09 20:55:55,189 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575 2023-02-09 20:55:55,191 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182 2023-02-09 20:55:55,194 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2 2023-02-09 20:55:55,196 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283 2023-02-09 20:55:55,196 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(310): Store=96a50bc53ff96fbc6fcb8cf6c4a41083/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:55,196 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,197 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2 2023-02-09 20:55:55,197 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2 2023-02-09 20:55:55,198 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96a50bc53ff96fbc6fcb8cf6c4a41083 columnFamilyName cf2 2023-02-09 20:55:55,201 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e 2023-02-09 20:55:55,203 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0 2023-02-09 20:55:55,206 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e 2023-02-09 20:55:55,208 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641 2023-02-09 20:55:55,210 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6 2023-02-09 20:55:55,212 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a 2023-02-09 20:55:55,214 DEBUG [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8 2023-02-09 20:55:55,214 INFO [StoreOpener-96a50bc53ff96fbc6fcb8cf6c4a41083-1] regionserver.HStore(310): Store=96a50bc53ff96fbc6fcb8cf6c4a41083/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:55,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:55,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 96a50bc53ff96fbc6fcb8cf6c4a41083; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72478914, jitterRate=0.0800199806690216}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:55,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 96a50bc53ff96fbc6fcb8cf6c4a41083: Running coprocessor pre-open hook at 1675976155175Writing region info on filesystem at 1675976155176 (+1 ms)Initializing all the Stores at 1675976155176Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976155176Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976155176Cleaning up temporary data from old regions at 1675976155216 (+40 ms)Cleaning up detritus from prior splits at 1675976155216Running coprocessor post-open hooks at 1675976155218 (+2 ms)Region opened successfully at 1675976155218 2023-02-09 20:55:55,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083., pid=133, masterSystemTime=1675976155172 2023-02-09 20:55:55,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:55,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): 96a50bc53ff96fbc6fcb8cf6c4a41083/cf1 is initiating minor compaction (all files) 2023-02-09 20:55:55,220 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): 96a50bc53ff96fbc6fcb8cf6c4a41083/cf2 is initiating minor compaction (all files) 2023-02-09 20:55:55,220 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 96a50bc53ff96fbc6fcb8cf6c4a41083/cf1 in testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,220 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 96a50bc53ff96fbc6fcb8cf6c4a41083/cf2 in testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,220 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/.tmp, totalSize=34.9 K 2023-02-09 20:55:55,220 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/.tmp, totalSize=34.9 K 2023-02-09 20:55:55,221 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976151913 2023-02-09 20:55:55,221 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976151913 2023-02-09 20:55:55,221 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976152086 2023-02-09 20:55:55,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,222 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976152260 2023-02-09 20:55:55,222 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976152086 2023-02-09 20:55:55,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d2502d975b11df7ff327f1cde3bb1527, NAME => 'testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.', STARTKEY => '71', ENDKEY => ''} 2023-02-09 20:55:55,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:55,222 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976152838 2023-02-09 20:55:55,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,222 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976152260 2023-02-09 20:55:55,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:55,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,222 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976153007 2023-02-09 20:55:55,223 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976152838 2023-02-09 20:55:55,223 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976153169 2023-02-09 20:55:55,223 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976153007 2023-02-09 20:55:55,223 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976153744 2023-02-09 20:55:55,223 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976153169 2023-02-09 20:55:55,224 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976153744 2023-02-09 20:55:55,224 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,225 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=96a50bc53ff96fbc6fcb8cf6c4a41083, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,225 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155225"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976155225"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976155225"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976155225"}]},"ts":"1675976155225"} 2023-02-09 20:55:55,226 DEBUG [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf1 2023-02-09 20:55:55,226 DEBUG [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf1 2023-02-09 20:55:55,226 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d2502d975b11df7ff327f1cde3bb1527 columnFamilyName cf1 2023-02-09 20:55:55,227 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] regionserver.HStore(310): Store=d2502d975b11df7ff327f1cde3bb1527/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:55,227 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=133, resume processing ppid=131 2023-02-09 20:55:55,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, ppid=131, state=SUCCESS; OpenRegionProcedure 96a50bc53ff96fbc6fcb8cf6c4a41083, server=jenkins-hbase9.apache.org,43365,1675976048828 in 207 msec 2023-02-09 20:55:55,228 DEBUG [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf2 2023-02-09 20:55:55,228 DEBUG [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf2 2023-02-09 20:55:55,229 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d2502d975b11df7ff327f1cde3bb1527 columnFamilyName cf2 2023-02-09 20:55:55,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, ASSIGN in 363 msec 2023-02-09 20:55:55,229 INFO [StoreOpener-d2502d975b11df7ff327f1cde3bb1527-1] regionserver.HStore(310): Store=d2502d975b11df7ff327f1cde3bb1527/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:55,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:55,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened d2502d975b11df7ff327f1cde3bb1527; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59566154, jitterRate=-0.11239513754844666}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:55,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for d2502d975b11df7ff327f1cde3bb1527: Running coprocessor pre-open hook at 1675976155223Writing region info on filesystem at 1675976155223Initializing all the Stores at 1675976155223Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976155223Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976155224 (+1 ms)Cleaning up temporary data from old regions at 1675976155231 (+7 ms)Cleaning up detritus from prior splits at 1675976155231Running coprocessor post-open hooks at 1675976155235 (+4 ms)Region opened successfully at 1675976155235 2023-02-09 20:55:55,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527., pid=134, masterSystemTime=1675976155172 2023-02-09 20:55:55,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,237 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=d2502d975b11df7ff327f1cde3bb1527, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,237 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155237"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976155237"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976155237"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976155237"}]},"ts":"1675976155237"} 2023-02-09 20:55:55,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=132 2023-02-09 20:55:55,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=132, state=SUCCESS; OpenRegionProcedure d2502d975b11df7ff327f1cde3bb1527, server=jenkins-hbase9.apache.org,43365,1675976048828 in 215 msec 2023-02-09 20:55:55,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=128 2023-02-09 20:55:55,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, ASSIGN in 375 msec 2023-02-09 20:55:55,243 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=eda24b137d6b92f849cd269a2ce19e3f, daughterA=96a50bc53ff96fbc6fcb8cf6c4a41083, daughterB=d2502d975b11df7ff327f1cde3bb1527 in 1.4920 sec 2023-02-09 20:55:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43365] ipc.CallRunner(144): callId: 987 service: ClientService methodName: Scan size: 157 connection: 172.31.2.10:56516 deadline: 1675976215252, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f. is not online on jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,255 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96a50bc53ff96fbc6fcb8cf6c4a41083#cf1#compaction#127 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:55,256 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): 96a50bc53ff96fbc6fcb8cf6c4a41083#cf2#compaction#126 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:55,267 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/.tmp/cf2/6856398b396d4611ae1abe7062c84c36 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/6856398b396d4611ae1abe7062c84c36 2023-02-09 20:55:55,269 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/.tmp/cf1/f1b5374e14094e5fab0719799aa57293 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/f1b5374e14094e5fab0719799aa57293 2023-02-09 20:55:55,271 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96a50bc53ff96fbc6fcb8cf6c4a41083/cf2 of 96a50bc53ff96fbc6fcb8cf6c4a41083 into 6856398b396d4611ae1abe7062c84c36(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:55,271 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 96a50bc53ff96fbc6fcb8cf6c4a41083: 2023-02-09 20:55:55,271 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083., storeName=96a50bc53ff96fbc6fcb8cf6c4a41083/cf2, priority=9, startTime=1675976155220; duration=0sec 2023-02-09 20:55:55,271 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:55,273 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96a50bc53ff96fbc6fcb8cf6c4a41083/cf1 of 96a50bc53ff96fbc6fcb8cf6c4a41083 into f1b5374e14094e5fab0719799aa57293(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:55,273 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 96a50bc53ff96fbc6fcb8cf6c4a41083: 2023-02-09 20:55:55,273 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083., storeName=96a50bc53ff96fbc6fcb8cf6c4a41083/cf1, priority=9, startTime=1675976155219; duration=0sec 2023-02-09 20:55:55,273 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:55,362 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '71', inclusive 2023-02-09 20:55:55,365 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:55,366 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=135, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-09 20:55:55,370 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976155369"}]},"ts":"1675976155369"} 2023-02-09 20:55:55,370 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLING in hbase:meta 2023-02-09 20:55:55,372 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionEmptyDaughter to state=DISABLING 2023-02-09 20:55:55,372 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, UNASSIGN}, {pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, UNASSIGN}] 2023-02-09 20:55:55,373 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, UNASSIGN 2023-02-09 20:55:55,374 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=d2502d975b11df7ff327f1cde3bb1527, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,374 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155374"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976155374"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976155374"}]},"ts":"1675976155374"} 2023-02-09 20:55:55,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure d2502d975b11df7ff327f1cde3bb1527, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:55,376 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, UNASSIGN 2023-02-09 20:55:55,377 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=96a50bc53ff96fbc6fcb8cf6c4a41083, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,377 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155377"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976155377"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976155377"}]},"ts":"1675976155377"} 2023-02-09 20:55:55,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=139, ppid=136, state=RUNNABLE; CloseRegionProcedure 96a50bc53ff96fbc6fcb8cf6c4a41083, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:55,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-09 20:55:55,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing d2502d975b11df7ff327f1cde3bb1527, disabling compactions & flushes 2023-02-09 20:55:55,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. after waiting 0 ms 2023-02-09 20:55:55,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,530 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=85 2023-02-09 20:55:55,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:55,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527. 2023-02-09 20:55:55,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for d2502d975b11df7ff327f1cde3bb1527: Waiting for close lock at 1675976155526Running coprocessor pre-close hooks at 1675976155526Disabling compacts and flushes for region at 1675976155526Disabling writes for close at 1675976155527 (+1 ms)Writing region close event to WAL at 1675976155528 (+1 ms)Running coprocessor post-close hooks at 1675976155531 (+3 ms)Closed at 1675976155531 2023-02-09 20:55:55,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 96a50bc53ff96fbc6fcb8cf6c4a41083, disabling compactions & flushes 2023-02-09 20:55:55,532 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. after waiting 0 ms 2023-02-09 20:55:55,532 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,532 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=d2502d975b11df7ff327f1cde3bb1527, regionState=CLOSED 2023-02-09 20:55:55,532 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155532"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976155532"}]},"ts":"1675976155532"} 2023-02-09 20:55:55,533 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283] to archive 2023-02-09 20:55:55,533 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:55,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=137 2023-02-09 20:55:55,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure d2502d975b11df7ff327f1cde3bb1527, server=jenkins-hbase9.apache.org,43365,1675976048828 in 159 msec 2023-02-09 20:55:55,535 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-78f0467acee341509b2723b1c92e2e26 2023-02-09 20:55:55,536 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=d2502d975b11df7ff327f1cde3bb1527, UNASSIGN in 163 msec 2023-02-09 20:55:55,536 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-95465d9994ed4cecb206f4d7dfd0e182 2023-02-09 20:55:55,538 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8962ff4db13c40698c6932eaee26e8ea 2023-02-09 20:55:55,539 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-8cd232e7eba44fcba61432d1936a4575 2023-02-09 20:55:55,540 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-a2b2c88ece9948d58e126ddf9c5430d2 2023-02-09 20:55:55,540 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-1f1834684b004d93ac1a6cc81f2d6a24 2023-02-09 20:55:55,541 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-df5b17ed570d41c99d11fa2c09f3e283 2023-02-09 20:55:55,542 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6] to archive 2023-02-09 20:55:55,543 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:55,544 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-37642d2ecf1f4ed59863121ceb8e65b0 2023-02-09 20:55:55,545 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-c068cac6558c434289b47c40f623b0c8 2023-02-09 20:55:55,546 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-2d3b3da6032846fb86fba36fee5a5b1e 2023-02-09 20:55:55,546 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-59d991d1fcd841f0bd3c6d13f9076641 2023-02-09 20:55:55,547 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-4f0051263d9e44b4bee77780a98cb22e 2023-02-09 20:55:55,548 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-b99447302f264b1582f1f32bc9e1bc1a 2023-02-09 20:55:55,549 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/testSplitTableRegionEmptyDaughter=eda24b137d6b92f849cd269a2ce19e3f-7b2833977cf546fabca7510f7c13aff6 2023-02-09 20:55:55,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:55:55,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:55,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083. 2023-02-09 20:55:55,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 96a50bc53ff96fbc6fcb8cf6c4a41083: Waiting for close lock at 1675976155532Running coprocessor pre-close hooks at 1675976155532Disabling compacts and flushes for region at 1675976155532Disabling writes for close at 1675976155532Writing region close event to WAL at 1675976155550 (+18 ms)Running coprocessor post-close hooks at 1675976155552 (+2 ms)Closed at 1675976155552 2023-02-09 20:55:55,553 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,553 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=96a50bc53ff96fbc6fcb8cf6c4a41083, regionState=CLOSED 2023-02-09 20:55:55,554 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1675976155553"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976155553"}]},"ts":"1675976155553"} 2023-02-09 20:55:55,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=136 2023-02-09 20:55:55,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=136, state=SUCCESS; CloseRegionProcedure 96a50bc53ff96fbc6fcb8cf6c4a41083, server=jenkins-hbase9.apache.org,43365,1675976048828 in 176 msec 2023-02-09 20:55:55,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=135 2023-02-09 20:55:55,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=96a50bc53ff96fbc6fcb8cf6c4a41083, UNASSIGN in 184 msec 2023-02-09 20:55:55,557 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976155557"}]},"ts":"1675976155557"} 2023-02-09 20:55:55,558 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLED in hbase:meta 2023-02-09 20:55:55,566 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionEmptyDaughter to state=DISABLED 2023-02-09 20:55:55,567 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionEmptyDaughter in 200 msec 2023-02-09 20:55:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-02-09 20:55:55,671 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 135 completed 2023-02-09 20:55:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-02-09 20:55:55,678 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:55,678 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,678 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,680 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/recovered.edits] 2023-02-09 20:55:55,680 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/recovered.edits] 2023-02-09 20:55:55,680 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/recovered.edits] 2023-02-09 20:55:55,684 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/f1b5374e14094e5fab0719799aa57293 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf1/f1b5374e14094e5fab0719799aa57293 2023-02-09 20:55:55,684 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/1f1834684b004d93ac1a6cc81f2d6a24 2023-02-09 20:55:55,685 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/78f0467acee341509b2723b1c92e2e26 2023-02-09 20:55:55,686 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/recovered.edits/88.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527/recovered.edits/88.seqid 2023-02-09 20:55:55,686 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8962ff4db13c40698c6932eaee26e8ea 2023-02-09 20:55:55,687 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/6856398b396d4611ae1abe7062c84c36 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/cf2/6856398b396d4611ae1abe7062c84c36 2023-02-09 20:55:55,687 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/d2502d975b11df7ff327f1cde3bb1527 2023-02-09 20:55:55,688 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/8cd232e7eba44fcba61432d1936a4575 2023-02-09 20:55:55,689 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/95465d9994ed4cecb206f4d7dfd0e182 2023-02-09 20:55:55,689 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083/recovered.edits/90.seqid 2023-02-09 20:55:55,690 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/96a50bc53ff96fbc6fcb8cf6c4a41083 2023-02-09 20:55:55,690 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/a2b2c88ece9948d58e126ddf9c5430d2 2023-02-09 20:55:55,691 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf1/df5b17ed570d41c99d11fa2c09f3e283 2023-02-09 20:55:55,693 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/2d3b3da6032846fb86fba36fee5a5b1e 2023-02-09 20:55:55,695 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/37642d2ecf1f4ed59863121ceb8e65b0 2023-02-09 20:55:55,696 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/4f0051263d9e44b4bee77780a98cb22e 2023-02-09 20:55:55,697 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/59d991d1fcd841f0bd3c6d13f9076641 2023-02-09 20:55:55,699 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/7b2833977cf546fabca7510f7c13aff6 2023-02-09 20:55:55,702 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/b99447302f264b1582f1f32bc9e1bc1a 2023-02-09 20:55:55,704 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/cf2/c068cac6558c434289b47c40f623b0c8 2023-02-09 20:55:55,706 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f/recovered.edits/85.seqid 2023-02-09 20:55:55,706 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegionEmptyDaughter/eda24b137d6b92f849cd269a2ce19e3f 2023-02-09 20:55:55,706 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-02-09 20:55:55,709 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,711 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionEmptyDaughter from hbase:meta 2023-02-09 20:55:55,713 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionEmptyDaughter' descriptor. 2023-02-09 20:55:55,715 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,715 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionEmptyDaughter' from region states. 2023-02-09 20:55:55,715 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976155715"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:55,715 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976155715"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:55,715 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976155715"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:55,717 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:55:55,717 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => eda24b137d6b92f849cd269a2ce19e3f, NAME => 'testSplitTableRegionEmptyDaughter,,1675976151410.eda24b137d6b92f849cd269a2ce19e3f.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 96a50bc53ff96fbc6fcb8cf6c4a41083, NAME => 'testSplitTableRegionEmptyDaughter,,1675976153750.96a50bc53ff96fbc6fcb8cf6c4a41083.', STARTKEY => '', ENDKEY => '71'}, {ENCODED => d2502d975b11df7ff327f1cde3bb1527, NAME => 'testSplitTableRegionEmptyDaughter,71,1675976153750.d2502d975b11df7ff327f1cde3bb1527.', STARTKEY => '71', ENDKEY => ''}] 2023-02-09 20:55:55,717 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionEmptyDaughter' as deleted. 2023-02-09 20:55:55,717 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976155717"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:55,718 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionEmptyDaughter state from META 2023-02-09 20:55:55,720 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-02-09 20:55:55,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter in 49 msec 2023-02-09 20:55:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-02-09 20:55:55,775 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 140 completed 2023-02-09 20:55:55,789 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=373 (was 376), OpenFileDescriptor=825 (was 829), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=118 (was 129), ProcessCount=168 (was 168), AvailableMemoryMB=10488 (was 10494) 2023-02-09 20:55:55,803 INFO [Listener at localhost/45551] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=373, OpenFileDescriptor=825, MaxFileDescriptor=60000, SystemLoadAverage=118, ProcessCount=168, AvailableMemoryMB=10487 2023-02-09 20:55:55,803 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:55,806 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-02-09 20:55:55,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(492): Client=jenkins//172.31.2.10 set balanceSwitch=false 2023-02-09 20:55:55,808 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegion 2023-02-09 20:55:55,809 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_PRE_OPERATION 2023-02-09 20:55:55,810 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-09 20:55:55,811 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:55,811 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff empty. 2023-02-09 20:55:55,811 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:55,812 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-02-09 20:55:55,822 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/.tabledesc/.tableinfo.0000000001 2023-02-09 20:55:55,823 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(7671): creating {ENCODED => 4b1d7da31ea85dc2e07c63180d5811ff, NAME => 'testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegion', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1603): Closing 4b1d7da31ea85dc2e07c63180d5811ff, disabling compactions & flushes 2023-02-09 20:55:55,831 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. after waiting 0 ms 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:55,831 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:55,831 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1557): Region close journal for 4b1d7da31ea85dc2e07c63180d5811ff: Waiting for close lock at 1675976155831Disabling compacts and flushes for region at 1675976155831Disabling writes for close at 1675976155831Writing region close event to WAL at 1675976155831Closed at 1675976155831 2023-02-09 20:55:55,833 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ADD_TO_META 2023-02-09 20:55:55,834 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976155834"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976155834"}]},"ts":"1675976155834"} 2023-02-09 20:55:55,835 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-09 20:55:55,835 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-09 20:55:55,836 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976155836"}]},"ts":"1675976155836"} 2023-02-09 20:55:55,836 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLING in hbase:meta 2023-02-09 20:55:55,839 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase9.apache.org=0} racks are {/default-rack=0} 2023-02-09 20:55:55,840 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-09 20:55:55,840 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-09 20:55:55,840 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-09 20:55:55,840 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-09 20:55:55,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, ASSIGN}] 2023-02-09 20:55:55,842 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, ASSIGN 2023-02-09 20:55:55,842 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:55,992 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-09 20:55:55,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=4b1d7da31ea85dc2e07c63180d5811ff, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:55,994 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976155993"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976155993"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976155993"}]},"ts":"1675976155993"} 2023-02-09 20:55:55,995 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure 4b1d7da31ea85dc2e07c63180d5811ff, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:56,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4b1d7da31ea85dc2e07c63180d5811ff, NAME => 'testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.', STARTKEY => '', ENDKEY => ''} 2023-02-09 20:55:56,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:56,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:56,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,151 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,152 DEBUG [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1 2023-02-09 20:55:56,152 DEBUG [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1 2023-02-09 20:55:56,153 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4b1d7da31ea85dc2e07c63180d5811ff columnFamilyName cf1 2023-02-09 20:55:56,153 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] regionserver.HStore(310): Store=4b1d7da31ea85dc2e07c63180d5811ff/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:56,153 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,154 DEBUG [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2 2023-02-09 20:55:56,154 DEBUG [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2 2023-02-09 20:55:56,155 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4b1d7da31ea85dc2e07c63180d5811ff columnFamilyName cf2 2023-02-09 20:55:56,155 INFO [StoreOpener-4b1d7da31ea85dc2e07c63180d5811ff-1] regionserver.HStore(310): Store=4b1d7da31ea85dc2e07c63180d5811ff/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:56,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:56,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:56,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-09 20:55:56,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 4b1d7da31ea85dc2e07c63180d5811ff; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67470715, jitterRate=0.005392000079154968}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:56,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 4b1d7da31ea85dc2e07c63180d5811ff: Running coprocessor pre-open hook at 1675976156150Writing region info on filesystem at 1675976156150Initializing all the Stores at 1675976156151 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976156151Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976156151Cleaning up temporary data from old regions at 1675976156157 (+6 ms)Cleaning up detritus from prior splits at 1675976156157Running coprocessor post-open hooks at 1675976156161 (+4 ms)Region opened successfully at 1675976156161 2023-02-09 20:55:56,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff., pid=143, masterSystemTime=1675976156146 2023-02-09 20:55:56,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,163 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=4b1d7da31ea85dc2e07c63180d5811ff, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,163 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976156163"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976156163"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976156163"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976156163"}]},"ts":"1675976156163"} 2023-02-09 20:55:56,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=142 2023-02-09 20:55:56,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure 4b1d7da31ea85dc2e07c63180d5811ff, server=jenkins-hbase9.apache.org,43365,1675976048828 in 169 msec 2023-02-09 20:55:56,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=141 2023-02-09 20:55:56,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, ASSIGN in 326 msec 2023-02-09 20:55:56,167 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-09 20:55:56,167 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976156167"}]},"ts":"1675976156167"} 2023-02-09 20:55:56,168 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLED in hbase:meta 2023-02-09 20:55:56,170 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_POST_OPERATION 2023-02-09 20:55:56,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testSplitTableRegion in 363 msec 2023-02-09 20:55:56,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:56,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:56,316 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:56,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:56,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:56,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,316 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,317 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,317 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,317 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,317 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,317 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,318 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,319 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,320 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:56,320 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:56,320 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,320 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,320 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,320 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:56,321 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,320 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,321 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,321 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=120 B heapSize=928 B 2023-02-09 20:55:56,321 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,322 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,322 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,324 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,330 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/d7d4b9446a4744d8883246abb24a0bbd 2023-02-09 20:55:56,341 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/86eee97875ac4af6bafe2883f7d18444 2023-02-09 20:55:56,344 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/d7d4b9446a4744d8883246abb24a0bbd as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd 2023-02-09 20:55:56,347 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:56,347 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/86eee97875ac4af6bafe2883f7d18444 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444 2023-02-09 20:55:56,350 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:55:56,351 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 30ms, sequenceid=6, compaction requested=false 2023-02-09 20:55:56,351 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegion' 2023-02-09 20:55:56,351 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:56,351 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,351 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:56,352 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:56,352 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,352 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,352 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,353 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,353 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,353 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,353 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30d8d8f3[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:56,355 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:56,355 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,356 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,357 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,357 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:56,356 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,357 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:56,356 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,357 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,357 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,357 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:56,358 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,358 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,358 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,358 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,362 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:56,365 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:56,365 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:56,365 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:56,365 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:56,365 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,366 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,367 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,367 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,368 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,369 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,369 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,370 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:56,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:56,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,372 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,373 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,442 ERROR [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid 5ff62a41-196c-4396-be17-ac730c5a50b4) service to localhost/127.0.0.1:38591 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:55:56,473 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:56,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:56,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:56,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:56,484 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:56,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:56,484 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:56,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,485 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,486 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,486 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,486 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:56,488 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:56,488 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,488 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:56,489 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:56,496 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,496 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,496 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,496 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,497 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,497 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,501 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/a841f22cc7ab4bdcb245579af4d34b12 2023-02-09 20:55:56,511 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a 2023-02-09 20:55:56,514 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/a841f22cc7ab4bdcb245579af4d34b12 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12 2023-02-09 20:55:56,517 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:56,517 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a 2023-02-09 20:55:56,521 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a, entries=10, sequenceid=19, filesize=5.0 K 2023-02-09 20:55:56,522 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 32ms, sequenceid=19, compaction requested=false 2023-02-09 20:55:56,522 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:56,522 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,523 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,523 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,524 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,524 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:56,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d3feda[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:56,525 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:56,525 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,527 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,527 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:56,530 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,530 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:56,533 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:56,533 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:56,533 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:56,533 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:56,533 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:56,534 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,534 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,534 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,535 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,535 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,535 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,536 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,538 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,538 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,538 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,541 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,541 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,542 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,642 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:56,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:56,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:56,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:56,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:56,654 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:56,654 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,654 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:56,654 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:56,654 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,654 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,655 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,655 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,656 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,656 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,657 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,658 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,658 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,658 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,658 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:56,660 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,667 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,667 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,667 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,668 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,668 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,670 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/5e5b45425928401f85bd9a4220609eab 2023-02-09 20:55:56,680 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/94632780d8c645099a1b52eb2f544c09 2023-02-09 20:55:56,683 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/5e5b45425928401f85bd9a4220609eab as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab 2023-02-09 20:55:56,686 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:56,686 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/94632780d8c645099a1b52eb2f544c09 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09 2023-02-09 20:55:56,689 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09, entries=10, sequenceid=32, filesize=5.0 K 2023-02-09 20:55:56,690 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 32ms, sequenceid=32, compaction requested=true 2023-02-09 20:55:56,690 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:56,690 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:56,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:56,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,690 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,691 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,691 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,691 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,692 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f7decf6[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:56,693 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:56,693 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,694 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,694 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:56,695 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:56,702 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,702 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:56,705 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:56,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:56,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:56,705 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:56,705 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,707 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,707 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,707 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,708 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,710 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,710 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,710 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,713 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,713 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,714 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,814 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:56,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:56,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:56,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:56,826 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:56,826 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,826 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:56,826 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:56,827 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,827 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,829 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,829 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,829 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,829 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:56,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:56,831 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:56,831 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,831 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:56,831 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:56,832 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,833 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,833 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,833 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,833 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,833 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,843 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/ea7c5d9774d74dcb8fea5f273346a5fb 2023-02-09 20:55:56,854 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/e75867be51894d3089f6e72c07ce38bd 2023-02-09 20:55:56,858 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/ea7c5d9774d74dcb8fea5f273346a5fb as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb 2023-02-09 20:55:56,861 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:56,861 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/e75867be51894d3089f6e72c07ce38bd as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd 2023-02-09 20:55:56,864 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd, entries=10, sequenceid=45, filesize=5.0 K 2023-02-09 20:55:56,865 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 34ms, sequenceid=45, compaction requested=true 2023-02-09 20:55:56,865 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:56,865 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:56,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:56,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:56,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:56,865 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:56,867 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,867 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7067b435[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:56,869 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:56,869 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:56,870 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:56,870 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:56,871 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:56,872 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,872 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,872 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,873 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,873 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:56,877 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:56,877 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:56,877 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:56,877 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:56,877 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:56,880 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,879 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,880 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,881 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:56,883 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:56,883 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,884 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:56,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:56,890 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:56,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,889 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,890 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:56,890 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:56,890 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:56,890 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:56,890 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:56,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:56,891 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:56,991 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:56,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:57,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:57,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:57,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:57,002 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:57,002 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,002 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:57,002 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:57,002 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,002 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,003 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,003 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,003 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,003 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,004 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,004 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:57,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,005 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,005 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,005 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,006 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,006 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,007 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,008 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,008 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,009 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,009 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,009 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,015 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/80577344192d4a039808e0d112c403c2 2023-02-09 20:55:57,023 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/f1c263b3dae04426ad7d5727d7c8dbde 2023-02-09 20:55:57,026 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/80577344192d4a039808e0d112c403c2 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2 2023-02-09 20:55:57,029 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:57,030 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/f1c263b3dae04426ad7d5727d7c8dbde as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde 2023-02-09 20:55:57,032 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde, entries=10, sequenceid=58, filesize=5.0 K 2023-02-09 20:55:57,033 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 27ms, sequenceid=58, compaction requested=true 2023-02-09 20:55:57,033 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:57,033 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,035 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,035 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,035 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,035 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ef58a[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:57,037 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:57,037 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,038 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,038 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:57,040 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,040 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,042 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:57,045 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:57,045 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:57,045 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:57,045 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:57,045 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,049 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,050 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,050 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,052 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,052 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,053 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,056 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:57,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:57,055 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,056 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,057 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,157 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:57,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:57,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-02-09 20:55:57,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-02-09 20:55:57,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-02-09 20:55:57,173 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-02-09 20:55:57,173 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,173 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-02-09 20:55:57,173 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-02-09 20:55:57,174 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,174 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,175 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,175 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,175 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,175 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,176 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,176 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-02-09 20:55:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,36111,1675976048792' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-02-09 20:55:57,177 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-02-09 20:55:57,178 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,178 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,178 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. started... 2023-02-09 20:55:57,178 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,39321,1675976048743' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,178 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-02-09 20:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,181 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,181 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,181 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,182 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,182 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,193 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/14b1e83eb2f24d99a4ecc5105fad22eb 2023-02-09 20:55:57,202 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/17a6b9fd5a4648eabb10c60dd3197010 2023-02-09 20:55:57,206 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/14b1e83eb2f24d99a4ecc5105fad22eb as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb 2023-02-09 20:55:57,209 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:57,209 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/17a6b9fd5a4648eabb10c60dd3197010 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010 2023-02-09 20:55:57,212 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010, entries=10, sequenceid=71, filesize=5.0 K 2023-02-09 20:55:57,212 INFO [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2947): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 34ms, sequenceid=71, compaction requested=true 2023-02-09 20:55:57,213 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] regionserver.HRegion(2445): Flush status journal for 4b1d7da31ea85dc2e07c63180d5811ff: 2023-02-09 20:55:57,213 DEBUG [rs(jenkins-hbase9.apache.org,43365,1675976048828)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,213 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-02-09 20:55:57,213 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-02-09 20:55:57,213 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,213 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-02-09 20:55:57,213 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-02-09 20:55:57,214 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,214 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,214 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,215 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-02-09 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,43365,1675976048828' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-02-09 20:55:57,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b76feaf[Count = 0] remaining members to acquire global barrier 2023-02-09 20:55:57,216 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-02-09 20:55:57,216 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,217 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,217 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,217 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,36111,1675976048792' in zk 2023-02-09 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,218 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,218 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,43365,1675976048828' in zk 2023-02-09 20:55:57,218 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase9.apache.org,39321,1675976048743' in zk 2023-02-09 20:55:57,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,227 DEBUG [member: 'jenkins-hbase9.apache.org,36111,1675976048792' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-02-09 20:55:57,228 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,39321,1675976048743' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,228 DEBUG [member: 'jenkins-hbase9.apache.org,43365,1675976048828' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-02-09 20:55:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase9.apache.org,43365,1675976048828': 2023-02-09 20:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,43365,1675976048828' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-02-09 20:55:57,231 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-02-09 20:55:57,231 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-02-09 20:55:57,231 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-02-09 20:55:57,231 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-02-09 20:55:57,231 INFO [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-02-09 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-02-09 20:55:57,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,233 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,234 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,235 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-02-09 20:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-02-09 20:55:57,236 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-02-09 20:55:57,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,237 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-02-09 20:55:57,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-02-09 20:55:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,241 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-02-09 20:55:57,242 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-02-09 20:55:57,241 DEBUG [(jenkins-hbase9.apache.org,42381,1675976047566)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-02-09 20:55:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-02-09 20:55:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-09 20:55:57,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-02-09 20:55:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-09 20:55:57,342 DEBUG [Listener at localhost/45551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-02-09 20:55:57,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-02-09 20:55:57,349 DEBUG [Listener at localhost/45551] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,350 DEBUG [Listener at localhost/45551] procedure2.ProcedureExecutor(1029): Stored pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=4b1d7da31ea85dc2e07c63180d5811ff, daughterA=4fd4a72843f9f10e7c1aa6b8ffe9acaa, daughterB=0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:57,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=4b1d7da31ea85dc2e07c63180d5811ff, daughterA=4fd4a72843f9f10e7c1aa6b8ffe9acaa, daughterB=0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:57,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=4b1d7da31ea85dc2e07c63180d5811ff, daughterA=4fd4a72843f9f10e7c1aa6b8ffe9acaa, daughterB=0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:57,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=4b1d7da31ea85dc2e07c63180d5811ff, daughterA=4fd4a72843f9f10e7c1aa6b8ffe9acaa, daughterB=0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:57,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, UNASSIGN}] 2023-02-09 20:55:57,353 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, UNASSIGN 2023-02-09 20:55:57,354 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=4b1d7da31ea85dc2e07c63180d5811ff, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:57,354 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976157354"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976157354"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976157354"}]},"ts":"1675976157354"} 2023-02-09 20:55:57,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=146, ppid=145, state=RUNNABLE; CloseRegionProcedure 4b1d7da31ea85dc2e07c63180d5811ff, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:57,438 ERROR [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa) service to localhost/127.0.0.1:38591 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-02-09 20:55:57,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 4b1d7da31ea85dc2e07c63180d5811ff, disabling compactions & flushes 2023-02-09 20:55:57,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. after waiting 0 ms 2023-02-09 20:55:57,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 4b1d7da31ea85dc2e07c63180d5811ff 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-02-09 20:55:57,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/a17d3ca71cb34dc185534749c3d55827 2023-02-09 20:55:57,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/08f476e313dc46dc9465a2c83729a3d8 2023-02-09 20:55:57,531 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf1/a17d3ca71cb34dc185534749c3d55827 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827 2023-02-09 20:55:57,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:57,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/.tmp/cf2/08f476e313dc46dc9465a2c83729a3d8 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8 2023-02-09 20:55:57,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8, entries=8, sequenceid=82, filesize=5.0 K 2023-02-09 20:55:57,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 32ms, sequenceid=82, compaction requested=true 2023-02-09 20:55:57,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-02-09 20:55:57,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:57,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. 2023-02-09 20:55:57,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 4b1d7da31ea85dc2e07c63180d5811ff: Waiting for close lock at 1675976157506Running coprocessor pre-close hooks at 1675976157506Disabling compacts and flushes for region at 1675976157506Disabling writes for close at 1675976157506Obtaining lock to block concurrent updates at 1675976157506Preparing flush snapshotting stores in 4b1d7da31ea85dc2e07c63180d5811ff at 1675976157506Finished memstore snapshotting testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff., syncing WAL and waiting on mvcc, flushsize=dataSize=480, getHeapSize=2144, getOffHeapSize=0, getCellsCount=16 at 1675976157507 (+1 ms)Flushing stores of testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. at 1675976157507Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf1: creating writer at 1675976157508 (+1 ms)Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf1: appending metadata at 1675976157510 (+2 ms)Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf1: closing flushed file at 1675976157510Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf2: creating writer at 1675976157522 (+12 ms)Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf2: appending metadata at 1675976157523 (+1 ms)Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf2: closing flushed file at 1675976157523Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf1: reopening flushed file at 1675976157532 (+9 ms)Flushing 4b1d7da31ea85dc2e07c63180d5811ff/cf2: reopening flushed file at 1675976157535 (+3 ms)Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 4b1d7da31ea85dc2e07c63180d5811ff in 32ms, sequenceid=82, compaction requested=true at 1675976157538 (+3 ms)Writing region close event to WAL at 1675976157540 (+2 ms)Running coprocessor post-close hooks at 1675976157543 (+3 ms)Closed at 1675976157543 2023-02-09 20:55:57,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,544 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=4b1d7da31ea85dc2e07c63180d5811ff, regionState=CLOSED 2023-02-09 20:55:57,544 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976157544"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976157544"}]},"ts":"1675976157544"} 2023-02-09 20:55:57,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=146, resume processing ppid=145 2023-02-09 20:55:57,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, ppid=145, state=SUCCESS; CloseRegionProcedure 4b1d7da31ea85dc2e07c63180d5811ff, server=jenkins-hbase9.apache.org,43365,1675976048828 in 190 msec 2023-02-09 20:55:57,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=144 2023-02-09 20:55:57,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=4b1d7da31ea85dc2e07c63180d5811ff, UNASSIGN in 194 msec 2023-02-09 20:55:57,556 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=144 splitting 14 storefiles, region=4b1d7da31ea85dc2e07c63180d5811ff, threads=14 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,556 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,557 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,558 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,626 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,637 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,637 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,638 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,642 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,642 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,644 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,645 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,645 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:57,645 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:58,045 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:58,046 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2 for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:58,047 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:58,379 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb for region: 4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:58,379 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=144 split storefiles for region 4b1d7da31ea85dc2e07c63180d5811ff Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-02-09 20:55:58,416 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:58,418 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-02-09 20:55:58,419 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1675976158419"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1675976158419"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1675976158419"}]},"ts":"1675976158419"} 2023-02-09 20:55:58,419 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976158419"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976158419"}]},"ts":"1675976158419"} 2023-02-09 20:55:58,419 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976158419"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976158419"}]},"ts":"1675976158419"} 2023-02-09 20:55:58,421 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, ASSIGN}, {pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, ASSIGN}] 2023-02-09 20:55:58,422 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, ASSIGN 2023-02-09 20:55:58,422 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:58,423 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, ASSIGN 2023-02-09 20:55:58,423 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase9.apache.org,43365,1675976048828; forceNewPlan=false, retain=false 2023-02-09 20:55:58,572 INFO [jenkins-hbase9:42381] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-02-09 20:55:58,574 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=4fd4a72843f9f10e7c1aa6b8ffe9acaa, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:58,574 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158574"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976158574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976158574"}]},"ts":"1675976158574"} 2023-02-09 20:55:58,575 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=147, state=RUNNABLE; OpenRegionProcedure 4fd4a72843f9f10e7c1aa6b8ffe9acaa, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:58,577 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=0bc1293d7e29aae8243cfa100eadc420, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:58,577 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158577"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976158577"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976158577"}]},"ts":"1675976158577"} 2023-02-09 20:55:58,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=150, ppid=148, state=RUNNABLE; OpenRegionProcedure 0bc1293d7e29aae8243cfa100eadc420, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:58,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 0bc1293d7e29aae8243cfa100eadc420, NAME => 'testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.', STARTKEY => '41', ENDKEY => ''} 2023-02-09 20:55:58,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:58,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:58,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,730 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,731 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1 2023-02-09 20:55:58,731 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1 2023-02-09 20:55:58,731 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0bc1293d7e29aae8243cfa100eadc420 columnFamilyName cf1 2023-02-09 20:55:58,742 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-top 2023-02-09 20:55:58,744 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-top 2023-02-09 20:55:58,747 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-top 2023-02-09 20:55:58,750 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-top 2023-02-09 20:55:58,753 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-top 2023-02-09 20:55:58,756 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-top 2023-02-09 20:55:58,758 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-top 2023-02-09 20:55:58,758 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(310): Store=0bc1293d7e29aae8243cfa100eadc420/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:58,758 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,759 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2 2023-02-09 20:55:58,759 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2 2023-02-09 20:55:58,759 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0bc1293d7e29aae8243cfa100eadc420 columnFamilyName cf2 2023-02-09 20:55:58,771 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-top 2023-02-09 20:55:58,774 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-top 2023-02-09 20:55:58,777 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-top 2023-02-09 20:55:58,780 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-top 2023-02-09 20:55:58,783 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-top 2023-02-09 20:55:58,786 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-top 2023-02-09 20:55:58,789 DEBUG [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-top 2023-02-09 20:55:58,790 INFO [StoreOpener-0bc1293d7e29aae8243cfa100eadc420-1] regionserver.HStore(310): Store=0bc1293d7e29aae8243cfa100eadc420/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:58,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:58,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:58,796 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 0bc1293d7e29aae8243cfa100eadc420; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59597909, jitterRate=-0.11192195117473602}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:58,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 0bc1293d7e29aae8243cfa100eadc420: Running coprocessor pre-open hook at 1675976158729Writing region info on filesystem at 1675976158730 (+1 ms)Initializing all the Stores at 1675976158730Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976158730Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976158730Cleaning up temporary data from old regions at 1675976158793 (+63 ms)Cleaning up detritus from prior splits at 1675976158793Running coprocessor post-open hooks at 1675976158796 (+3 ms)Region opened successfully at 1675976158796 2023-02-09 20:55:58,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420., pid=150, masterSystemTime=1675976158726 2023-02-09 20:55:58,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,798 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:58,798 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:58,798 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,798 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): 0bc1293d7e29aae8243cfa100eadc420/cf2 is initiating minor compaction (all files) 2023-02-09 20:55:58,798 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,798 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 0bc1293d7e29aae8243cfa100eadc420/cf2 in testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,798 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): 0bc1293d7e29aae8243cfa100eadc420/cf1 is initiating minor compaction (all files) 2023-02-09 20:55:58,798 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 0bc1293d7e29aae8243cfa100eadc420/cf1 in testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,799 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/.tmp, totalSize=34.9 K 2023-02-09 20:55:58,799 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-top] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/.tmp, totalSize=34.9 K 2023-02-09 20:55:58,799 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976156310 2023-02-09 20:55:58,799 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1675976156310 2023-02-09 20:55:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:58,800 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976156474 2023-02-09 20:55:58,800 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1675976156474 2023-02-09 20:55:58,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,800 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=0bc1293d7e29aae8243cfa100eadc420, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:58,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 4fd4a72843f9f10e7c1aa6b8ffe9acaa, NAME => 'testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.', STARTKEY => '', ENDKEY => '41'} 2023-02-09 20:55:58,800 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158800"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976158800"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976158800"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976158800"}]},"ts":"1675976158800"} 2023-02-09 20:55:58,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-02-09 20:55:58,800 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976156644 2023-02-09 20:55:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(865): Instantiated testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-09 20:55:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7890): checking encryption for 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7893): checking classloading for 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,801 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1675976156644 2023-02-09 20:55:58,801 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976156816 2023-02-09 20:55:58,801 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1675976156816 2023-02-09 20:55:58,802 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976156992 2023-02-09 20:55:58,802 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1675976156992 2023-02-09 20:55:58,802 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,802 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976157158 2023-02-09 20:55:58,803 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1675976157158 2023-02-09 20:55:58,803 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976157343 2023-02-09 20:55:58,803 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-09 20:55:58,803 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; OpenRegionProcedure 0bc1293d7e29aae8243cfa100eadc420, server=jenkins-hbase9.apache.org,43365,1675976048828 in 224 msec 2023-02-09 20:55:58,803 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1 2023-02-09 20:55:58,803 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1 2023-02-09 20:55:58,804 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1675976157343 2023-02-09 20:55:58,804 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4fd4a72843f9f10e7c1aa6b8ffe9acaa columnFamilyName cf1 2023-02-09 20:55:58,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, ASSIGN in 382 msec 2023-02-09 20:55:58,817 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): 0bc1293d7e29aae8243cfa100eadc420#cf2#compaction#142 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:58,818 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0bc1293d7e29aae8243cfa100eadc420#cf1#compaction#143 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:58,819 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-bottom 2023-02-09 20:55:58,828 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-bottom 2023-02-09 20:55:58,831 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-bottom 2023-02-09 20:55:58,834 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-bottom 2023-02-09 20:55:58,835 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/.tmp/cf1/5e27b5c0af3641b0adfdbd4edb8d4404 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e27b5c0af3641b0adfdbd4edb8d4404 2023-02-09 20:55:58,835 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/.tmp/cf2/7ad85918f59d4754a8bfc172540b73cb as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/7ad85918f59d4754a8bfc172540b73cb 2023-02-09 20:55:58,837 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-bottom 2023-02-09 20:55:58,839 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0bc1293d7e29aae8243cfa100eadc420/cf1 of 0bc1293d7e29aae8243cfa100eadc420 into 5e27b5c0af3641b0adfdbd4edb8d4404(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:58,839 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 0bc1293d7e29aae8243cfa100eadc420: 2023-02-09 20:55:58,839 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0bc1293d7e29aae8243cfa100eadc420/cf2 of 0bc1293d7e29aae8243cfa100eadc420 into 7ad85918f59d4754a8bfc172540b73cb(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:58,839 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420., storeName=0bc1293d7e29aae8243cfa100eadc420/cf1, priority=9, startTime=1675976158797; duration=0sec 2023-02-09 20:55:58,839 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 0bc1293d7e29aae8243cfa100eadc420: 2023-02-09 20:55:58,839 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420., storeName=0bc1293d7e29aae8243cfa100eadc420/cf2, priority=9, startTime=1675976158797; duration=0sec 2023-02-09 20:55:58,839 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,839 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,839 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-bottom 2023-02-09 20:55:58,842 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-bottom 2023-02-09 20:55:58,842 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(310): Store=4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:58,842 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,843 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2 2023-02-09 20:55:58,843 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2 2023-02-09 20:55:58,843 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4fd4a72843f9f10e7c1aa6b8ffe9acaa columnFamilyName cf2 2023-02-09 20:55:58,852 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-bottom 2023-02-09 20:55:58,855 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-bottom 2023-02-09 20:55:58,857 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-bottom 2023-02-09 20:55:58,859 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-bottom 2023-02-09 20:55:58,862 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-bottom 2023-02-09 20:55:58,864 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-bottom 2023-02-09 20:55:58,867 DEBUG [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(539): loaded hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-bottom 2023-02-09 20:55:58,867 INFO [StoreOpener-4fd4a72843f9f10e7c1aa6b8ffe9acaa-1] regionserver.HStore(310): Store=4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-09 20:55:58,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-02-09 20:55:58,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1054): writing seq id for 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:58,870 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1071): Opened 4fd4a72843f9f10e7c1aa6b8ffe9acaa; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66912180, jitterRate=-0.0029308199882507324}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-02-09 20:55:58,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(964): Region open journal for 4fd4a72843f9f10e7c1aa6b8ffe9acaa: Running coprocessor pre-open hook at 1675976158801Writing region info on filesystem at 1675976158801Initializing all the Stores at 1675976158802 (+1 ms)Instantiating store for column family {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976158802Instantiating store for column family {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1675976158802Cleaning up temporary data from old regions at 1675976158869 (+67 ms)Cleaning up detritus from prior splits at 1675976158869Running coprocessor post-open hooks at 1675976158870 (+1 ms)Region opened successfully at 1675976158870 2023-02-09 20:55:58,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa., pid=149, masterSystemTime=1675976158726 2023-02-09 20:55:58,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-02-09 20:55:58,872 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:58,872 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-02-09 20:55:58,872 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,872 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1912): 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2 is initiating minor compaction (all files) 2023-02-09 20:55:58,872 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,872 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1912): 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1 is initiating minor compaction (all files) 2023-02-09 20:55:58,872 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2258): Starting compaction of 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2 in testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,872 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2258): Starting compaction of 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1 in testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,873 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/.tmp, totalSize=34.9 K 2023-02-09 20:55:58,873 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-bottom] into tmpdir=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/.tmp, totalSize=34.9 K 2023-02-09 20:55:58,873 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976156310 2023-02-09 20:55:58,873 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1675976156310 2023-02-09 20:55:58,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976156474 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1675976156474 2023-02-09 20:55:58,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=4fd4a72843f9f10e7c1aa6b8ffe9acaa, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976156644 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1675976156644 2023-02-09 20:55:58,874 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976158874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1675976158874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1675976158874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1675976158874"}]},"ts":"1675976158874"} 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976156816 2023-02-09 20:55:58,874 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1675976156816 2023-02-09 20:55:58,875 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976156992 2023-02-09 20:55:58,875 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1675976156992 2023-02-09 20:55:58,875 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976157158 2023-02-09 20:55:58,875 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting 14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1675976157158 2023-02-09 20:55:58,875 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] compactions.Compactor(207): Compacting 08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976157343 2023-02-09 20:55:58,876 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] compactions.Compactor(207): Compacting a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1675976157343 2023-02-09 20:55:58,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=149, resume processing ppid=147 2023-02-09 20:55:58,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=147, state=SUCCESS; OpenRegionProcedure 4fd4a72843f9f10e7c1aa6b8ffe9acaa, server=jenkins-hbase9.apache.org,43365,1675976048828 in 300 msec 2023-02-09 20:55:58,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=147, resume processing ppid=144 2023-02-09 20:55:58,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, ASSIGN in 456 msec 2023-02-09 20:55:58,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegion, parent=4b1d7da31ea85dc2e07c63180d5811ff, daughterA=4fd4a72843f9f10e7c1aa6b8ffe9acaa, daughterB=0bc1293d7e29aae8243cfa100eadc420 in 1.5290 sec 2023-02-09 20:55:58,885 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] throttle.PressureAwareThroughputController(145): 4fd4a72843f9f10e7c1aa6b8ffe9acaa#cf2#compaction#144 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:58,885 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4fd4a72843f9f10e7c1aa6b8ffe9acaa#cf1#compaction#145 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-02-09 20:55:58,894 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/.tmp/cf1/2cdec44636e24287b234253340b74d23 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/2cdec44636e24287b234253340b74d23 2023-02-09 20:55:58,894 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/.tmp/cf2/bff9447fb2794c159a86546a7a89eb50 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/bff9447fb2794c159a86546a7a89eb50 2023-02-09 20:55:58,897 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2 of 4fd4a72843f9f10e7c1aa6b8ffe9acaa into bff9447fb2794c159a86546a7a89eb50(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:58,897 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1 of 4fd4a72843f9f10e7c1aa6b8ffe9acaa into 2cdec44636e24287b234253340b74d23(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-02-09 20:55:58,897 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.HRegion(2288): Compaction status journal for 4fd4a72843f9f10e7c1aa6b8ffe9acaa: 2023-02-09 20:55:58,897 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.HRegion(2288): Compaction status journal for 4fd4a72843f9f10e7c1aa6b8ffe9acaa: 2023-02-09 20:55:58,897 INFO [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa., storeName=4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2, priority=9, startTime=1675976158872; duration=0sec 2023-02-09 20:55:58,897 INFO [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa., storeName=4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1, priority=9, startTime=1675976158871; duration=0sec 2023-02-09 20:55:58,897 DEBUG [RS:2;jenkins-hbase9:43365-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:58,897 DEBUG [RS:2;jenkins-hbase9:43365-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-02-09 20:55:59,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43365] ipc.CallRunner(144): callId: 1116 service: ClientService methodName: Scan size: 143 connection: 172.31.2.10:56516 deadline: 1675976219102, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff. is not online on jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:59,211 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:55:59,216 INFO [Listener at localhost/45551] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-02-09 20:55:59,218 DEBUG [Listener at localhost/45551] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-02-09 20:55:59,224 WARN [Listener at localhost/45551] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-02-09 20:55:59,225 INFO [Listener at localhost/45551] client.HBaseAdmin$15(890): Started disable of testSplitTableRegion 2023-02-09 20:55:59,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$11(2405): Client=jenkins//172.31.2.10 disable testSplitTableRegion 2023-02-09 20:55:59,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=151, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-09 20:55:59,229 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976159229"}]},"ts":"1675976159229"} 2023-02-09 20:55:59,230 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLING in hbase:meta 2023-02-09 20:55:59,231 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegion to state=DISABLING 2023-02-09 20:55:59,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, UNASSIGN}, {pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, UNASSIGN}] 2023-02-09 20:55:59,232 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, UNASSIGN 2023-02-09 20:55:59,232 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=0bc1293d7e29aae8243cfa100eadc420, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:59,233 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976159232"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976159232"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976159232"}]},"ts":"1675976159232"} 2023-02-09 20:55:59,233 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=154, ppid=153, state=RUNNABLE; CloseRegionProcedure 0bc1293d7e29aae8243cfa100eadc420, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:59,235 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, UNASSIGN 2023-02-09 20:55:59,235 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=4fd4a72843f9f10e7c1aa6b8ffe9acaa, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:59,235 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976159235"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1675976159235"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1675976159235"}]},"ts":"1675976159235"} 2023-02-09 20:55:59,238 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; CloseRegionProcedure 4fd4a72843f9f10e7c1aa6b8ffe9acaa, server=jenkins-hbase9.apache.org,43365,1675976048828}] 2023-02-09 20:55:59,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-09 20:55:59,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:59,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 0bc1293d7e29aae8243cfa100eadc420, disabling compactions & flushes 2023-02-09 20:55:59,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:59,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:59,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. after waiting 0 ms 2023-02-09 20:55:59,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:59,386 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-top] to archive 2023-02-09 20:55:59,386 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:59,388 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,389 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,390 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,391 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,391 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,392 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,393 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,394 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-top, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-top] to archive 2023-02-09 20:55:59,395 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:59,396 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,397 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,398 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,399 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,400 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,401 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,402 DEBUG [StoreCloser-testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:55:59,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:59,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420. 2023-02-09 20:55:59,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 0bc1293d7e29aae8243cfa100eadc420: Waiting for close lock at 1675976159385Running coprocessor pre-close hooks at 1675976159385Disabling compacts and flushes for region at 1675976159385Disabling writes for close at 1675976159385Writing region close event to WAL at 1675976159403 (+18 ms)Running coprocessor post-close hooks at 1675976159405 (+2 ms)Closed at 1675976159405 2023-02-09 20:55:59,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:59,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:59,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 4fd4a72843f9f10e7c1aa6b8ffe9acaa, disabling compactions & flushes 2023-02-09 20:55:59,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:59,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:59,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. after waiting 0 ms 2023-02-09 20:55:59,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:59,407 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=0bc1293d7e29aae8243cfa100eadc420, regionState=CLOSED 2023-02-09 20:55:59,407 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976159407"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976159407"}]},"ts":"1675976159407"} 2023-02-09 20:55:59,408 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827-bottom] to archive 2023-02-09 20:55:59,408 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:59,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=153 2023-02-09 20:55:59,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=153, state=SUCCESS; CloseRegionProcedure 0bc1293d7e29aae8243cfa100eadc420, server=jenkins-hbase9.apache.org,43365,1675976048828 in 175 msec 2023-02-09 20:55:59,410 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/d7d4b9446a4744d8883246abb24a0bbd.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=0bc1293d7e29aae8243cfa100eadc420, UNASSIGN in 177 msec 2023-02-09 20:55:59,411 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a841f22cc7ab4bdcb245579af4d34b12.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,412 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/5e5b45425928401f85bd9a4220609eab.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,412 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/ea7c5d9774d74dcb8fea5f273346a5fb.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,413 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/80577344192d4a039808e0d112c403c2.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,414 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/14b1e83eb2f24d99a4ecc5105fad22eb.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,415 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/a17d3ca71cb34dc185534749c3d55827.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,416 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010-bottom, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff->hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8-bottom] to archive 2023-02-09 20:55:59,417 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-02-09 20:55:59,418 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/86eee97875ac4af6bafe2883f7d18444.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,419 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,420 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/94632780d8c645099a1b52eb2f544c09.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,420 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/e75867be51894d3089f6e72c07ce38bd.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,421 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/f1c263b3dae04426ad7d5727d7c8dbde.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,422 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/17a6b9fd5a4648eabb10c60dd3197010.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,423 DEBUG [StoreCloser-testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/08f476e313dc46dc9465a2c83729a3d8.4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-02-09 20:55:59,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:55:59,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa. 2023-02-09 20:55:59,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 4fd4a72843f9f10e7c1aa6b8ffe9acaa: Waiting for close lock at 1675976159407Running coprocessor pre-close hooks at 1675976159407Disabling compacts and flushes for region at 1675976159407Disabling writes for close at 1675976159407Writing region close event to WAL at 1675976159424 (+17 ms)Running coprocessor post-close hooks at 1675976159426 (+2 ms)Closed at 1675976159426 2023-02-09 20:55:59,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:59,427 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=4fd4a72843f9f10e7c1aa6b8ffe9acaa, regionState=CLOSED 2023-02-09 20:55:59,427 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1675976159427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1675976159427"}]},"ts":"1675976159427"} 2023-02-09 20:55:59,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-02-09 20:55:59,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; CloseRegionProcedure 4fd4a72843f9f10e7c1aa6b8ffe9acaa, server=jenkins-hbase9.apache.org,43365,1675976048828 in 192 msec 2023-02-09 20:55:59,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=152, resume processing ppid=151 2023-02-09 20:55:59,430 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=4fd4a72843f9f10e7c1aa6b8ffe9acaa, UNASSIGN in 197 msec 2023-02-09 20:55:59,431 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1675976159431"}]},"ts":"1675976159431"} 2023-02-09 20:55:59,431 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLED in hbase:meta 2023-02-09 20:55:59,433 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegion to state=DISABLED 2023-02-09 20:55:59,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, state=SUCCESS; DisableTableProcedure table=testSplitTableRegion in 207 msec 2023-02-09 20:55:59,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-02-09 20:55:59,530 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegion, procId: 151 completed 2023-02-09 20:55:59,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.HMaster$5(2215): Client=jenkins//172.31.2.10 delete testSplitTableRegion 2023-02-09 20:55:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] procedure2.ProcedureExecutor(1029): Stored pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,533 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,534 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=156, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-02-09 20:55:59,537 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,537 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:59,537 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:59,539 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/recovered.edits] 2023-02-09 20:55:59,539 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/recovered.edits] 2023-02-09 20:55:59,539 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2, FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/recovered.edits] 2023-02-09 20:55:59,541 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e27b5c0af3641b0adfdbd4edb8d4404 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf1/5e27b5c0af3641b0adfdbd4edb8d4404 2023-02-09 20:55:59,542 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/2cdec44636e24287b234253340b74d23 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf1/2cdec44636e24287b234253340b74d23 2023-02-09 20:55:59,542 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/14b1e83eb2f24d99a4ecc5105fad22eb 2023-02-09 20:55:59,544 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/5e5b45425928401f85bd9a4220609eab 2023-02-09 20:55:59,544 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/7ad85918f59d4754a8bfc172540b73cb to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/cf2/7ad85918f59d4754a8bfc172540b73cb 2023-02-09 20:55:59,544 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/bff9447fb2794c159a86546a7a89eb50 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/cf2/bff9447fb2794c159a86546a7a89eb50 2023-02-09 20:55:59,545 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/80577344192d4a039808e0d112c403c2 2023-02-09 20:55:59,546 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a17d3ca71cb34dc185534749c3d55827 2023-02-09 20:55:59,546 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420/recovered.edits/90.seqid 2023-02-09 20:55:59,546 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/recovered.edits/90.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa/recovered.edits/90.seqid 2023-02-09 20:55:59,547 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/0bc1293d7e29aae8243cfa100eadc420 2023-02-09 20:55:59,547 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4fd4a72843f9f10e7c1aa6b8ffe9acaa 2023-02-09 20:55:59,547 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/a841f22cc7ab4bdcb245579af4d34b12 2023-02-09 20:55:59,548 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/d7d4b9446a4744d8883246abb24a0bbd 2023-02-09 20:55:59,549 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf1/ea7c5d9774d74dcb8fea5f273346a5fb 2023-02-09 20:55:59,551 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/08f476e313dc46dc9465a2c83729a3d8 2023-02-09 20:55:59,552 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/17a6b9fd5a4648eabb10c60dd3197010 2023-02-09 20:55:59,552 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/86eee97875ac4af6bafe2883f7d18444 2023-02-09 20:55:59,553 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/91fafdc26acd4a8fb0bc7bb8fa763d7a 2023-02-09 20:55:59,554 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09 to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/94632780d8c645099a1b52eb2f544c09 2023-02-09 20:55:59,555 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/e75867be51894d3089f6e72c07ce38bd 2023-02-09 20:55:59,556 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/cf2/f1c263b3dae04426ad7d5727d7c8dbde 2023-02-09 20:55:59,558 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/recovered.edits/85.seqid to hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/archive/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff/recovered.edits/85.seqid 2023-02-09 20:55:59,558 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/.tmp/data/default/testSplitTableRegion/4b1d7da31ea85dc2e07c63180d5811ff 2023-02-09 20:55:59,558 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-02-09 20:55:59,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=156, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,561 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegion from hbase:meta 2023-02-09 20:55:59,562 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegion' descriptor. 2023-02-09 20:55:59,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=156, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegion' from region states. 2023-02-09 20:55:59,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976159563"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:59,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976159563"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:59,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1675976159563"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:59,565 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-02-09 20:55:59,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4b1d7da31ea85dc2e07c63180d5811ff, NAME => 'testSplitTableRegion,,1675976155807.4b1d7da31ea85dc2e07c63180d5811ff.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 4fd4a72843f9f10e7c1aa6b8ffe9acaa, NAME => 'testSplitTableRegion,,1675976157349.4fd4a72843f9f10e7c1aa6b8ffe9acaa.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 0bc1293d7e29aae8243cfa100eadc420, NAME => 'testSplitTableRegion,41,1675976157349.0bc1293d7e29aae8243cfa100eadc420.', STARTKEY => '41', ENDKEY => ''}] 2023-02-09 20:55:59,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegion' as deleted. 2023-02-09 20:55:59,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1675976159565"}]},"ts":"9223372036854775807"} 2023-02-09 20:55:59,566 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegion state from META 2023-02-09 20:55:59,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=156, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-02-09 20:55:59,568 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegion in 37 msec 2023-02-09 20:55:59,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42381] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-02-09 20:55:59,635 INFO [Listener at localhost/45551] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegion, procId: 156 completed 2023-02-09 20:55:59,648 INFO [Listener at localhost/45551] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=374 (was 373) - Thread LEAK? -, OpenFileDescriptor=816 (was 825), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=109 (was 118), ProcessCount=168 (was 168), AvailableMemoryMB=10476 (was 10487) 2023-02-09 20:55:59,648 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-09 20:55:59,649 INFO [Listener at localhost/45551] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-09 20:55:59,652 DEBUG [Listener at localhost/45551] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37e887de to 127.0.0.1:58959 2023-02-09 20:55:59,652 DEBUG [Listener at localhost/45551] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,677 DEBUG [Listener at localhost/45551] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-09 20:55:59,677 DEBUG [Listener at localhost/45551] util.JVMClusterUtil(257): Found active master hash=1891261643, stopped=false 2023-02-09 20:55:59,677 INFO [Listener at localhost/45551] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:55:59,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 20:55:59,679 INFO [Listener at localhost/45551] procedure2.ProcedureExecutor(629): Stopping 2023-02-09 20:55:59,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 20:55:59,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 20:55:59,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-09 20:55:59,679 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:55:59,680 DEBUG [Listener at localhost/45551] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x240f9ec7 to 127.0.0.1:58959 2023-02-09 20:55:59,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:55:59,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:55:59,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:55:59,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-09 20:55:59,680 DEBUG [Listener at localhost/45551] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,39321,1675976048743' ***** 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,36111,1675976048792' ***** 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,43365,1675976048828' ***** 2023-02-09 20:55:59,680 INFO [Listener at localhost/45551] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-09 20:55:59,681 INFO [RS:1;jenkins-hbase9:36111] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 20:55:59,681 INFO [RS:1;jenkins-hbase9:36111] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 20:55:59,681 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 20:55:59,681 INFO [RS:1;jenkins-hbase9:36111] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 20:55:59,681 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:59,681 DEBUG [RS:1;jenkins-hbase9:36111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ceeaf1 to 127.0.0.1:58959 2023-02-09 20:55:59,681 DEBUG [RS:1;jenkins-hbase9:36111] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,681 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,36111,1675976048792; all regions closed. 2023-02-09 20:55:59,681 INFO [RS:0;jenkins-hbase9:39321] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 20:55:59,681 INFO [RS:2;jenkins-hbase9:43365] regionserver.HeapMemoryManager(220): Stopping 2023-02-09 20:55:59,681 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 20:55:59,682 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-09 20:55:59,681 INFO [RS:0;jenkins-hbase9:39321] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 20:55:59,682 INFO [RS:2;jenkins-hbase9:43365] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-09 20:55:59,682 INFO [RS:0;jenkins-hbase9:39321] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 20:55:59,682 INFO [RS:2;jenkins-hbase9:43365] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-09 20:55:59,682 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(3304): Received CLOSE for 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:55:59,682 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:59,682 DEBUG [RS:0;jenkins-hbase9:39321] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x766fd7fc to 127.0.0.1:58959 2023-02-09 20:55:59,682 DEBUG [RS:0;jenkins-hbase9:39321] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,682 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:55:59,682 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,39321,1675976048743; all regions closed. 2023-02-09 20:55:59,682 DEBUG [RS:2;jenkins-hbase9:43365] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79cb1bdf to 127.0.0.1:58959 2023-02-09 20:55:59,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 9a9aa832e601f0693eaa144f8209e596, disabling compactions & flushes 2023-02-09 20:55:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:55:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:55:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. after waiting 0 ms 2023-02-09 20:55:59,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:55:59,682 DEBUG [RS:2;jenkins-hbase9:43365] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 9a9aa832e601f0693eaa144f8209e596 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-09 20:55:59,683 INFO [RS:2;jenkins-hbase9:43365] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 20:55:59,683 INFO [RS:2;jenkins-hbase9:43365] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 20:55:59,683 INFO [RS:2;jenkins-hbase9:43365] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 20:55:59,683 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-09 20:55:59,684 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-09 20:55:59,684 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1479): Online Regions={9a9aa832e601f0693eaa144f8209e596=hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596., 1588230740=hbase:meta,,1.1588230740} 2023-02-09 20:55:59,684 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-09 20:55:59,684 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-09 20:55:59,684 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-09 20:55:59,684 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-09 20:55:59,684 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-09 20:55:59,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=79.40 KB heapSize=125.77 KB 2023-02-09 20:55:59,685 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1505): Waiting on 1588230740, 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:55:59,695 DEBUG [RS:1;jenkins-hbase9:36111] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs 2023-02-09 20:55:59,695 INFO [RS:1;jenkins-hbase9:36111] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C36111%2C1675976048792:(num 1675976050421) 2023-02-09 20:55:59,695 DEBUG [RS:1;jenkins-hbase9:36111] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,695 INFO [RS:1;jenkins-hbase9:36111] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:55:59,696 DEBUG [RS:0;jenkins-hbase9:39321] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs 2023-02-09 20:55:59,696 INFO [RS:0;jenkins-hbase9:39321] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C39321%2C1675976048743:(num 1675976050420) 2023-02-09 20:55:59,696 DEBUG [RS:0;jenkins-hbase9:39321] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:55:59,696 INFO [RS:0;jenkins-hbase9:39321] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:55:59,855 INFO [RS:0;jenkins-hbase9:39321] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-09 20:55:59,855 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:55:59,855 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:55:59,855 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:55:59,856 INFO [RS:1;jenkins-hbase9:36111] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-09 20:55:59,857 INFO [RS:1;jenkins-hbase9:36111] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 20:55:59,857 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 20:55:59,857 INFO [RS:1;jenkins-hbase9:36111] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 20:55:59,857 INFO [RS:1;jenkins-hbase9:36111] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 20:55:59,858 INFO [RS:0;jenkins-hbase9:39321] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-09 20:55:59,858 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 20:55:59,858 INFO [RS:0;jenkins-hbase9:39321] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-09 20:55:59,858 INFO [RS:0;jenkins-hbase9:39321] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-09 20:55:59,858 INFO [RS:1;jenkins-hbase9:36111] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:36111 2023-02-09 20:55:59,859 INFO [RS:0;jenkins-hbase9:39321] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:39321 2023-02-09 20:55:59,864 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,39321,1675976048743 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:55:59,877 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:55:59,881 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@58eb2727 rejected from java.util.concurrent.ThreadPoolExecutor@5cd4b8e2[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:55:59,881 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:55:59,882 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2956cdf7 rejected from java.util.concurrent.ThreadPoolExecutor@5cd4b8e2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:55:59,882 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,39321,1675976048743] 2023-02-09 20:55:59,882 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,39321,1675976048743; numProcessing=1 2023-02-09 20:55:59,883 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:59,883 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:55:59,883 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,36111,1675976048792 2023-02-09 20:55:59,883 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12c501b9 rejected from java.util.concurrent.ThreadPoolExecutor@16fa8739[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:55:59,885 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,39321,1675976048743 already deleted, retry=false 2023-02-09 20:55:59,885 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,39321,1675976048743 expired; onlineServers=2 2023-02-09 20:55:59,886 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,36111,1675976048792] 2023-02-09 20:55:59,886 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,36111,1675976048792; numProcessing=2 2023-02-09 20:55:59,886 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1505): Waiting on 1588230740, 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:55:59,887 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,36111,1675976048792 already deleted, retry=false 2023-02-09 20:55:59,887 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,36111,1675976048792 expired; onlineServers=1 2023-02-09 20:55:59,888 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.29 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/info/f436f4ea3305458da90ec0ae468027d9 2023-02-09 20:55:59,891 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f436f4ea3305458da90ec0ae468027d9 2023-02-09 20:55:59,905 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.58 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/rep_barrier/4dd1cd9ed8224c3f9e161e2feb2a5dd4 2023-02-09 20:55:59,908 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4dd1cd9ed8224c3f9e161e2feb2a5dd4 2023-02-09 20:55:59,983 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:55:59,983 INFO [RS:0;jenkins-hbase9:39321] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,39321,1675976048743; zookeeper connection closed. 2023-02-09 20:55:59,983 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@516fabaf rejected from java.util.concurrent.ThreadPoolExecutor@5cd4b8e2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:55:59,983 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:39321-0x1003addd87e0001, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:55:59,983 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@66df91ee rejected from java.util.concurrent.ThreadPoolExecutor@5cd4b8e2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:55:59,983 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@19a4ce0e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@19a4ce0e 2023-02-09 20:56:00,083 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,083 INFO [RS:1;jenkins-hbase9:36111] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,36111,1675976048792; zookeeper connection closed. 2023-02-09 20:56:00,083 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5813c510 rejected from java.util.concurrent.ThreadPoolExecutor@16fa8739[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,083 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:36111-0x1003addd87e0002, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,083 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20e04f48 rejected from java.util.concurrent.ThreadPoolExecutor@16fa8739[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,083 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5c5f708e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5c5f708e 2023-02-09 20:56:00,086 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1505): Waiting on 1588230740, 9a9aa832e601f0693eaa144f8209e596 2023-02-09 20:56:00,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/.tmp/info/451bc5caffde48028b8a4ada5fc64384 2023-02-09 20:56:00,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/.tmp/info/451bc5caffde48028b8a4ada5fc64384 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/info/451bc5caffde48028b8a4ada5fc64384 2023-02-09 20:56:00,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/info/451bc5caffde48028b8a4ada5fc64384, entries=2, sequenceid=6, filesize=4.8 K 2023-02-09 20:56:00,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9a9aa832e601f0693eaa144f8209e596 in 590ms, sequenceid=6, compaction requested=false 2023-02-09 20:56:00,274 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-09 20:56:00,274 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-09 20:56:00,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/namespace/9a9aa832e601f0693eaa144f8209e596/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-09 20:56:00,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:56:00,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:56:00,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 9a9aa832e601f0693eaa144f8209e596: Waiting for close lock at 1675976159682Running coprocessor pre-close hooks at 1675976159682Disabling compacts and flushes for region at 1675976159682Disabling writes for close at 1675976159683 (+1 ms)Obtaining lock to block concurrent updates at 1675976159683Preparing flush snapshotting stores in 9a9aa832e601f0693eaa144f8209e596 at 1675976159683Finished memstore snapshotting hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1675976159683Flushing stores of hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. at 1675976159684 (+1 ms)Flushing 9a9aa832e601f0693eaa144f8209e596/info: creating writer at 1675976159684Flushing 9a9aa832e601f0693eaa144f8209e596/info: appending metadata at 1675976159689 (+5 ms)Flushing 9a9aa832e601f0693eaa144f8209e596/info: closing flushed file at 1675976159689Flushing 9a9aa832e601f0693eaa144f8209e596/info: reopening flushed file at 1675976160270 (+581 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 9a9aa832e601f0693eaa144f8209e596 in 590ms, sequenceid=6, compaction requested=false at 1675976160273 (+3 ms)Writing region close event to WAL at 1675976160274 (+1 ms)Running coprocessor post-close hooks at 1675976160277 (+3 ms)Closed at 1675976160278 (+1 ms) 2023-02-09 20:56:00,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1675976050870.9a9aa832e601f0693eaa144f8209e596. 2023-02-09 20:56:00,287 DEBUG [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-09 20:56:00,317 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.53 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/table/49e9236cd7ba415ba07cbb5459419f28 2023-02-09 20:56:00,320 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49e9236cd7ba415ba07cbb5459419f28 2023-02-09 20:56:00,320 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/info/f436f4ea3305458da90ec0ae468027d9 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/info/f436f4ea3305458da90ec0ae468027d9 2023-02-09 20:56:00,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f436f4ea3305458da90ec0ae468027d9 2023-02-09 20:56:00,323 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/info/f436f4ea3305458da90ec0ae468027d9, entries=59, sequenceid=210, filesize=11.1 K 2023-02-09 20:56:00,324 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/rep_barrier/4dd1cd9ed8224c3f9e161e2feb2a5dd4 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/rep_barrier/4dd1cd9ed8224c3f9e161e2feb2a5dd4 2023-02-09 20:56:00,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4dd1cd9ed8224c3f9e161e2feb2a5dd4 2023-02-09 20:56:00,327 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/rep_barrier/4dd1cd9ed8224c3f9e161e2feb2a5dd4, entries=24, sequenceid=210, filesize=7.5 K 2023-02-09 20:56:00,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/.tmp/table/49e9236cd7ba415ba07cbb5459419f28 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/table/49e9236cd7ba415ba07cbb5459419f28 2023-02-09 20:56:00,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49e9236cd7ba415ba07cbb5459419f28 2023-02-09 20:56:00,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/table/49e9236cd7ba415ba07cbb5459419f28, entries=36, sequenceid=210, filesize=8.0 K 2023-02-09 20:56:00,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 646ms, sequenceid=210, compaction requested=false 2023-02-09 20:56:00,334 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/data/hbase/meta/1588230740/recovered.edits/213.seqid, newMaxSeqId=213, maxSeqId=1 2023-02-09 20:56:00,334 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-02-09 20:56:00,334 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-09 20:56:00,335 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-09 20:56:00,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1675976159684Running coprocessor pre-close hooks at 1675976159684Disabling compacts and flushes for region at 1675976159684Disabling writes for close at 1675976159684Obtaining lock to block concurrent updates at 1675976159685 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1675976159685Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=81308, getHeapSize=128744, getOffHeapSize=0, getCellsCount=617 at 1675976159685Flushing stores of hbase:meta,,1.1588230740 at 1675976159690 (+5 ms)Flushing 1588230740/info: creating writer at 1675976159690Flushing 1588230740/info: appending metadata at 1675976159697 (+7 ms)Flushing 1588230740/info: closing flushed file at 1675976159697Flushing 1588230740/rep_barrier: creating writer at 1675976159892 (+195 ms)Flushing 1588230740/rep_barrier: appending metadata at 1675976159894 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1675976159895 (+1 ms)Flushing 1588230740/table: creating writer at 1675976159909 (+14 ms)Flushing 1588230740/table: appending metadata at 1675976159911 (+2 ms)Flushing 1588230740/table: closing flushed file at 1675976159911Flushing 1588230740/info: reopening flushed file at 1675976160321 (+410 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1675976160324 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1675976160327 (+3 ms)Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 646ms, sequenceid=210, compaction requested=false at 1675976160330 (+3 ms)Writing region close event to WAL at 1675976160332 (+2 ms)Running coprocessor post-close hooks at 1675976160334 (+2 ms)Closed at 1675976160335 (+1 ms) 2023-02-09 20:56:00,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-09 20:56:00,487 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,43365,1675976048828; all regions closed. 2023-02-09 20:56:00,491 DEBUG [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs 2023-02-09 20:56:00,491 INFO [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C43365%2C1675976048828.meta:.meta(num 1675976050544) 2023-02-09 20:56:00,495 DEBUG [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/oldWALs 2023-02-09 20:56:00,496 INFO [RS:2;jenkins-hbase9:43365] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C43365%2C1675976048828:(num 1675976050420) 2023-02-09 20:56:00,496 DEBUG [RS:2;jenkins-hbase9:43365] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:56:00,496 INFO [RS:2;jenkins-hbase9:43365] regionserver.LeaseManager(133): Closed leases 2023-02-09 20:56:00,496 INFO [RS:2;jenkins-hbase9:43365] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-09 20:56:00,496 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 20:56:00,497 INFO [RS:2;jenkins-hbase9:43365] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:43365 2023-02-09 20:56:00,498 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,43365,1675976048828 2023-02-09 20:56:00,498 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-09 20:56:00,498 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ac04ae5 rejected from java.util.concurrent.ThreadPoolExecutor@1effdd65[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,501 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,43365,1675976048828] 2023-02-09 20:56:00,501 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,43365,1675976048828; numProcessing=3 2023-02-09 20:56:00,502 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,43365,1675976048828 already deleted, retry=false 2023-02-09 20:56:00,502 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,43365,1675976048828 expired; onlineServers=0 2023-02-09 20:56:00,502 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,42381,1675976047566' ***** 2023-02-09 20:56:00,502 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-09 20:56:00,502 DEBUG [M:0;jenkins-hbase9:42381] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@318756f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,42381,1675976047566 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,42381,1675976047566; all regions closed. 2023-02-09 20:56:00,503 DEBUG [M:0;jenkins-hbase9:42381] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-09 20:56:00,503 DEBUG [M:0;jenkins-hbase9:42381] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-09 20:56:00,503 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-09 20:56:00,503 DEBUG [M:0;jenkins-hbase9:42381] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-09 20:56:00,503 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1675976050042] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1675976050042,5,FailOnTimeoutGroup] 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-09 20:56:00,503 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1675976050040] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1675976050040,5,FailOnTimeoutGroup] 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-02-09 20:56:00,503 DEBUG [M:0;jenkins-hbase9:42381] master.HMaster(1502): Stopping service threads 2023-02-09 20:56:00,503 INFO [M:0;jenkins-hbase9:42381] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-09 20:56:00,504 ERROR [M:0;jenkins-hbase9:42381] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] Thread[HFileArchiver-14,5,PEWorkerGroup] Thread[HFileArchiver-15,5,PEWorkerGroup] Thread[HFileArchiver-16,5,PEWorkerGroup] 2023-02-09 20:56:00,504 INFO [M:0;jenkins-hbase9:42381] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-09 20:56:00,504 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-09 20:56:00,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-09 20:56:00,504 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-09 20:56:00,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-09 20:56:00,504 DEBUG [M:0;jenkins-hbase9:42381] zookeeper.ZKUtil(398): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-09 20:56:00,504 WARN [M:0;jenkins-hbase9:42381] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-09 20:56:00,504 INFO [M:0;jenkins-hbase9:42381] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-09 20:56:00,505 INFO [M:0;jenkins-hbase9:42381] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-09 20:56:00,505 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-09 20:56:00,505 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:56:00,505 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:56:00,505 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-09 20:56:00,505 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:56:00,505 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=608.20 KB heapSize=719.10 KB 2023-02-09 20:56:00,516 INFO [M:0;jenkins-hbase9:42381] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=608.20 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b5db380e5bbd4bbebaa677f795c329b3 2023-02-09 20:56:00,519 INFO [M:0;jenkins-hbase9:42381] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b5db380e5bbd4bbebaa677f795c329b3 2023-02-09 20:56:00,519 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b5db380e5bbd4bbebaa677f795c329b3 as hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b5db380e5bbd4bbebaa677f795c329b3 2023-02-09 20:56:00,522 INFO [M:0;jenkins-hbase9:42381] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b5db380e5bbd4bbebaa677f795c329b3 2023-02-09 20:56:00,522 INFO [M:0;jenkins-hbase9:42381] regionserver.HStore(1080): Added hdfs://localhost:38591/user/jenkins/test-data/bcb76243-251c-ba40-a1fd-91599e9b6686/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b5db380e5bbd4bbebaa677f795c329b3, entries=156, sequenceid=1282, filesize=34.9 K 2023-02-09 20:56:00,523 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegion(2947): Finished flush of dataSize ~608.20 KB/622801, heapSize ~719.09 KB/736344, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 18ms, sequenceid=1282, compaction requested=false 2023-02-09 20:56:00,523 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-09 20:56:00,523 DEBUG [M:0;jenkins-hbase9:42381] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1675976160505Disabling compacts and flushes for region at 1675976160505Disabling writes for close at 1675976160505Obtaining lock to block concurrent updates at 1675976160505Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1675976160505Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=622801, getHeapSize=736344, getOffHeapSize=0, getCellsCount=1499 at 1675976160505Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1675976160505Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1675976160505Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1675976160508 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1675976160508Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1675976160520 (+12 ms)Finished flush of dataSize ~608.20 KB/622801, heapSize ~719.09 KB/736344, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 18ms, sequenceid=1282, compaction requested=false at 1675976160523 (+3 ms)Writing region close event to WAL at 1675976160523Closed at 1675976160523 2023-02-09 20:56:00,527 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-09 20:56:00,527 INFO [M:0;jenkins-hbase9:42381] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-09 20:56:00,527 INFO [M:0;jenkins-hbase9:42381] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:42381 2023-02-09 20:56:00,529 DEBUG [M:0;jenkins-hbase9:42381] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,42381,1675976047566 already deleted, retry=false 2023-02-09 20:56:00,601 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,601 INFO [RS:2;jenkins-hbase9:43365] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,43365,1675976048828; zookeeper connection closed. 2023-02-09 20:56:00,601 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7263f2aa rejected from java.util.concurrent.ThreadPoolExecutor@1effdd65[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,601 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): regionserver:43365-0x1003addd87e0003, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,601 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6feb3fb7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6feb3fb7 2023-02-09 20:56:00,601 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a18f5bc rejected from java.util.concurrent.ThreadPoolExecutor@1effdd65[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,601 INFO [Listener at localhost/45551] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-09 20:56:00,701 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,701 INFO [M:0;jenkins-hbase9:42381] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,42381,1675976047566; zookeeper connection closed. 2023-02-09 20:56:00,701 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@386b0e02 rejected from java.util.concurrent.ThreadPoolExecutor@22e3d4c3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 753] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,701 DEBUG [Listener at localhost/45551-EventThread] zookeeper.ZKWatcher(600): master:42381-0x1003addd87e0000, quorum=127.0.0.1:58959, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-09 20:56:00,701 ERROR [Listener at localhost/45551-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65651859 rejected from java.util.concurrent.ThreadPoolExecutor@22e3d4c3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 753] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-09 20:56:00,703 WARN [Listener at localhost/45551] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 20:56:00,705 INFO [Listener at localhost/45551] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 20:56:00,814 WARN [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-09 20:56:00,815 WARN [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid a0ea1a13-8490-4cba-a134-da32a3f13324) service to localhost/127.0.0.1:38591 2023-02-09 20:56:00,817 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data5/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:00,818 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data6/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:00,819 WARN [Listener at localhost/45551] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 20:56:00,824 INFO [Listener at localhost/45551] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 20:56:00,927 WARN [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid a9840449-fcf5-4f0a-a1ff-d40b0a43e2fa) service to localhost/127.0.0.1:38591 2023-02-09 20:56:00,927 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data3/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:00,928 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data4/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:00,929 WARN [Listener at localhost/45551] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-09 20:56:00,930 INFO [Listener at localhost/45551] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 20:56:01,032 WARN [BP-1574329966-172.31.2.10-1675976044485 heartbeating to localhost/127.0.0.1:38591] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1574329966-172.31.2.10-1675976044485 (Datanode Uuid 5ff62a41-196c-4396-be17-ac730c5a50b4) service to localhost/127.0.0.1:38591 2023-02-09 20:56:01,033 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data1/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:01,033 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/fa9592a8-e8c8-451d-5b48-e19b63c83936/cluster_a2248509-af8b-52cc-e048-ffe453238dca/dfs/data/data2/current/BP-1574329966-172.31.2.10-1675976044485] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-09 20:56:01,054 INFO [Listener at localhost/45551] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-09 20:56:01,245 INFO [Listener at localhost/45551] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-09 20:56:01,472 INFO [Listener at localhost/45551] hbase.HBaseTestingUtility(1293): Minicluster is down